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Not a week passes by without some research 

paper, feature article or product marketing mak- 
ing exaggerated or even entirely unlikely claims 

about the capabilities of Artificial Intelligence 

(AT). From academic papers that claim A\ can pre- 
dict criminality, personality or sexual orientation, 
to the companies that sell these supposed capa- 
bilities to law enforcement, border control or 
human resources departments around the world, 
fake and deeply flawed Al is rampant. 


The current amount of public interest in Al was 
spurred by the genuinely remarkable progress 
that has been made with some Al techniques in 
the past decade. For narrowly defined tasks, such 
as recognising objects, /\l is now able to perform 
at the same level or even better than humans. 
However, that progress, as Arvind Narayanan has 
argued, does not automatically translate into 

= solving other tasks. In fact, when it comes 

<= to predicting any social outcome, using Al is 

@ \ fundamentally dubious.’ 
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The ease and frequency with which Al’s real and 
imagined gains are conflated results in real, tan- 

gible harms. For those subject to automated sys- 

tems, it can mean the difference between getting 

ajob and not getting ajob, between being allowed 

to cross a border and being denied access. Worse, 

the ways in which these systems are so often | 
built in practice means that the burden of proof 

often falls on those affected to prove that they 

are in fact who they say they are. On a societal 

level, widespread belief in fake Al means that we 

risk redirecting resources to the wrong places. 

As Aidan Peppin argues in this book, it could 
also mean that public resistance to the tech- 
nology will end up stifling progress in areas 
where genuine progress is being made. 















What makes the phenomenon of 
fake Al especially curious is the 
fact that, in many ways, 2020-21 
has been a time of great /\| dis- 
illusionment. The Economist 4% Fr jananyA. (0019) 
dedicated its entire summer How to recognize AI snake oil. 
Princeton University, Department 

Technology Quarterly - the of Computer Science. https://www. 
Issue, concluding that “An cs.princeton.edu/~arvindn/talks/MIT- 
understanding of Al’s limi- |iaaueameemue ae 
tations is starting to sink Wem wmeiimesor 
a 2 understanding of AI’s limitations is 
mM: Fora technology that starting to sink in. The Economist. 
has been touted as the https://www.economist.com/ 
solution to virtually ever technology-quarterly/2020/06/11/ 

: . y Y an-understanding-of-ais-limitations- 
challengeimaginable—from (@@@nnrscscnven 
curing cancer, to fighting 


poverty, predicting 


criminality, reversing climate change and 
even ending death—Al has played a remark- 
ably minor role? in the global response to 
a very real challenge the world is facing 
today, the Covid-19 pandemic.‘ As we find 
ourselves on the downward slope of the 
Al hype cycle, this is a unique moment to 
take stock, to look back and to examine 
the underlying causes, dynamics, and log- 
ics behind the rise and fall of fake AI. 
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Bringing together differ- 
ent perspectives and voices 
from across disciplines and 
countries, this book 
interrogates 





3. Mateos-Garcia, J., Klinger, J., 
Stathoulopoulos, K. (2020) Artificial 

Intelligence and the Fight Against COVID- 

19. Nesta. https://www.nesta.org.uk/report/ 
artificial-intelligence-and-fight-against-covid-19/ 


4. Peach, K. (2020) How the pandemic has exposed 
Al’s limitations. Nesta. https://www.nesta.org.uk/blog/ 
how-the-pandemic-has-exposed-ais-limitations/ 













the rise and fall of Al hype, » X 


pseudoscience, and snake 
oil. It does this by drawing connec- 
tions between specific injustices inflicted 
by inappropriate A\, unpacking lazy and 
harmful assumptions made by developers 
when designing /\| tools and systems, and 
examining the existential underpinnings of 
the technology itself to ask: why are there 
so many useless, and even dangerously 
flawed, Al systems? 





















Any serious writing 
about AI will have 











to wrestle with 
the fact that Al 
itself has become 
an elusive term. 
As every computer 
scientist will be quick to 
point out, Al is an umbrella 
term that’s used for a set of 
related technologies. Yet while 
these same computer scientists 
are quick to offer a precise defi- 
nition and remind us that much 
of what we call Al today is in fact 
machine learning, in the pub- 
lic imagination, the term Al has 
taken on a meaning of its own. 
Here, Alis a catch-all phrase used 
to describe a wide-ranging set of 
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technologies, most of which apply statistical 
modelling to find patterns in large data sets and 
make predictions based on those patterns—as 
Fieke Jansen and Corinne Cath argue in their 
piece about the false hope that’s placed 
in Al registers. 









Just as Al has become an 
imprecise word, hype, pseu- @ 
doscience, and snake oil are 
frequently used interchangea- 

bly to call out Al research or £4! tools 

that claim to do something they either 
cannot, or should not do. If we look more 
closely however, these terms are distinct. Each 
highlights a different aspect of the phenom- 
enon that this book interrogates. 













As Abeba Birhane powerfully argues in her 
essay, Cheap Al, the return of pseudoscience, 
such as race science, is neither unique nor dis- 
tinct to AJ research. What is unique is that dusty 
and long discredited ideas have found new 
legitimacy through M\. Dangerously, 
they’ve acquired a veneer of innovation, 
a sheen of progress, even. By contrast, 
in a wide-ranging interview that con- 
siders how much, and how little, has 
changed since his original talk three 
years ago, Arvind Narayanan hones in 
on “i snake oil”, 
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expl2ining how it is distinct from pseudoscience. 
Vendors of /\i snake oil use deceptive marketing, 
fraud, and even scams to sell their products as 
solutions to problems for which Al techniques are 
either ill-equipped or completely useless. 


. 
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The environment in 
which snake oil 
and pseudosci- 
ence thrives is 
characterised 
by genuine 
excitement, 
unchallenged 
hype, bombas- 
tic headlines, 
and billions of 
dollars of invest- 
ment, all coupled with a naive belief 
in the idea that technology will save us. 
Journalist James Vincent writes about his 
first encounter with a PR pitch for an Al 
toothbrush and reflects on the challenges 
of covering hyped technology without fur- 
ther feeding unrealistic expectations. As 
someone who used to work as a content 
moderator for Google in the mid 2010s, 
Andrew Strait makes a plea against plac- 
ing too much hope on automation in 
content moderation. 





















\ Each piece in this book provides a differ- 
- ent perspective and proposes different 

answers to problems which circle around 

the shared question of what is driving exag- 
gerated, flawed or entirely unfounded hopes 
and expectations about AI. Against broad-brush 
claims, they call for precise thinking and scru- 
pulous expression. 





For Deborah Raji the lack of care with which 
engineers so often design algorithmic systems 
today belongs to a long history of engineering 
irresponsibility in constructing material arte- 
facts like bridges and cars. Razvan Amironesei, 
Emily Denton, Alex Hanna, Andrew Smart and 
Hilary Nicole describe how benchmark data- 
sets contribute to the belief that algorithmic 
systems are objective or scientific in nature. 
The artist Adam Harvey picks apart what exactly 
defines a “face” for, Al. 


A recurring theme throughout this book 
is that harms and risks are unevenly dis- 
tributed. Tulsi Parida and Aparna Ashok 
consider the effects of /\ inappropriately 
applied through the Indian concept of 
jugaad. Favour Borokini and Ridwan 
Oloyede warn of the dangers that come 
with [i hype in Nigeria’s fintech sector. © 


Amidst this feverishly hyped 
atmosphere, this book makes 
the case for nuance. It invites 
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readers to carefully separate the real pro- 2. 
gress that Al research has made in the 

past few years from fundamentally dubi- 

ous or dangerously exaggerated claims 
about .Al’s capabilities. We are not heading 
towards Artificial General Intelligence (AGI). We 
are not locked in an ff race that can only be won 
by those countries with the least regulation and 
the most investment. Instead, the real advances 
in AJ pose both old and new challenges that can 
only be tamed if we see AV for what it is. Namely, 
a powerful technology that at present is pro- | 
duced by only a handful of companies with 
workforces that are not representative of 
those who are disproportionately affected 
by its risks and harms. 











Frederike Kaltheuner is a tech policy 
consultant and researcher. She is the 
Director of the European AI Fund, 

a philanthropic initiative to strengthen 
civil society. 
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The term “snake oil” originates from the United 
States in the mid 19th century when Chinese 
immigrants working on the railroads introduced 
their American counterparts to a traditional treat- 
ment for arthritis and bursitis made of oil derived 
from the Chinese water snake. The effectiveness 
of the oil, which is high in omega-3 acids, and its 
subsequent popularity prompted some profiteers 
to get in on a lucrative market. These unscru- 
pulous sellers peddled quack remedies which 
contained inferior rattlesnake oil or completely 
arbitrary ingredients to an unsuspecting public. 
By the early 20th century, “snake oil” had taken 
on its modern, pejorative meaning to become a 
byword for fake miracle cures, groundless claims, 
and brazen falsehoods. 





Much of what is sold commercially as AI is snake 
oil, says Arvind Narayanan, Associate Professor 
for Computer Science at Princeton University— 
we have no evidence that it works, and based on 
our scientific understanding, we have strong rea- 
sons to believe that it couldn’t possibly work. And 
| yet, companies continue to market AI products 
that claim to predict anything from crime, to job 
performance, sexual orientation or gender. What 
makes the public so susceptible to these claims 1s 
the fact that in recent years, in some domains of AI 
research, there has been genuine and impressive 
progress. How, then, did AI become attached 
to so many products and services of ques- 
tionable or unverifiable quality, and slim to 
non-existent usefulness? 









Frederike Kaltheuner spoke to Arvind 
Narayanan via Zoom in January 2021. 
Frederike, from lockdown in Berlin, and 
Arvind from his office in Princeton. 













FK Your talk, How to Recognise Al 
Snake Oil went viral in 2019. What 
inspired you to write about A\ snake oil, 
and were you surprised by the amount 
of attention your talk received? 





AN Throughout the last 15 years or so 
of my research, one of my regular | 
motivations for getting straight into a 
research topic is when there is hype in the 
industry around something. That’s how | 
first got started on privacy research. My exper- 
tise, the desire for consumer protection, and 
the sense that industry hype got out of control 
all converged in the case of Al snake oil. The /{] 
narrative had been getting somewhat unhinged 
from reality for years, but the last straw was 
seeing how prominent these Al-based hiring 
companies have become. How many custom- 
ers they have, and how many millions of people 
have been put through these demeaning video 
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interviews where A] would supposedly 
figure out someone’s job suitability based 
on how they talked and other irrelevant 
factors. That’s really what triggered me 
to feel “I have to say something here”. 


| was very surprised by its reception. In 
addition to the attention on Twitter, | 
received something like 50 invita- 
tions for papers, books... That had 
never happened to me before. In 
retrospect | think many people sus- 
pected what was happening was 
snake oil but didn’t feel they had the 
expertise or authority to say anything. 
People were speaking up of course, 
but perhaps weren’t being taken as 
seriously because they didn’t have the 
“Professor of Computer Science” title. 
That we still put so much stock in cre- 
dentials is, | think, unfortunate. So when 
I stood up and said this, | was seen as 
someone who had the authority. People 
really felt it was an important counter to 
the hype. 





















EK: and it is still important to counter the 

hype today, especially in policy circles. 

Just how much of what is usually referred to as 

Al falls under the category of él snake oil? And 
how can we recognise it? 
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AN Much of 


what is sold commercially today as 
» “Al” is what I call “snake oil”. We have no 
evidence that it works, and based on 
our scientific understanding of the rel- 
evant domains, we have strong reasons 
to believe that it couldn’t possibly work. 
My educated guess is because “/\|” is a 
very loose umbrella term. This happens 
with buzzwords in the tech industry 
(like “blockchain”). After a point nobody 
~ really knows what it means. Some are 
not snake oil. There has been genuinely 
remarkable scientific progress. But because 
of this, companies put all kinds of systems 
under the A’ umbrella—including those you 
would have more accurately called regression 
20 years ago, or statistics, except that sta- 
tistics asks rigorous questions about whether 
something is working and how we can quan- 
tify it. But because of the hype, people have 
skipped this step and the public and policy- 
makers have bought into it. 









Surveys show that the public largely seems to 
believe that Artificial General Intelligence (AGI) 
is right around the corner—which would be a 
turning point in the history of human civilisation! 
| don’t think that’s true at all, and most experts 
don’t either. The idea that our current progress 
with Al would lead to AGI is as absurd as building 
a taller and taller ladder that reached the moon. 
There are fundamental differences between what 
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we're building now and what it would take to build 
AGI. AGI is not task-specific, so that’s in part why 
I think it will take something fundamentally new 
and different to get there. 


LDF 0.0631 


FK To build on your metaphor—if the genuinely 

remarkable scientific progress under the Al 

umbrella is a ladder to the moon, then AGI would 

take an entirely different ladder altogether. AI 

companies are pointing at genuine progress 

to make claims that require an entirely dif- 
ferent kind of progress altogether? 
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AN Right. There’s this massive confu- 
sion around what Al is, which com- 
panies have exploited to create hype. 
Point number two is that the types of 
applications of so-called “/\|” are funda- 
mentally dubious. One important cat- 
egory is predicting the future, that is, 
predicting social outcomes. Which 
kids might drop out of school? Who 
might be arrested for a crime in the 
future? Who should we hire? These 
are all contingent on an incredible 
array of factors that we still have 
trouble quantifying—and it’s not 
clear if we ever will. 
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A few scientific studies have looked 
rigorously at how good we are at pre- 
dicting these future social outcomes 
and shown that it’s barely better than 
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random. We can’t really do much better than 
simple regression models with a few variables. 
My favourite example is the “Fragile Families 
Challenge” led by my Princeton colleague Pro- 
fessor Matt Salganik, along with colleagues 
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and collaborators around the world. 
Hundreds of participants used state- 
of-the-art machine learning techniques 
¢ anda phenomenal dataset to scrutinise 
= “at-risk” kids over a decade and try to 
A predict (based on a child’s circums- 


tances today) what their outcomes * 


might be six years in the future. <.. 
The negative results are very tell- pe 4 
ing. No team, on any of these social Oe= 
outcomes, could produce predictions 

that were significantly better than 

random prediction. This is a powerful statement 
about why trying to predict future 

social outcomes is a fundamentally 

different type of task to those that Al 


has excelled at. These things don’t work well and 
we shouldn’t expect them to. 
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F K Which domains seem to have a lot of snake 
oil in them and why? 


AN My educated guess is that to understand 
the prevalence of Al snake oil it’s better to 
look at the consumers/buyers than the sellers. 
Companies will spring up around any type of 
technology for which high Gemana exists. 


So why are _ 
ICE AND HYPE 4 
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people willing to buy 
certain types of snake 
oil? That’s interesting. \ 
I think it’s because 
certain domains (like 
hiring) are so broken 
that even an elabo- 
rate random-number 
generator (which is 
what I think some 

of these \ tools are), 
is an improvement 
over what people are 
doing today. And | don’t 
make this statement lightly. 
In a domain like hiring we— 
culturally as well as in business— 
have a hard time admitting that 
there is not much we can do to 
predict who’s going to be most pro- 
ductive in a job. The best we can do 
is have some basic tests of preparation, 
ability and competence, and beyond that just 
accept that it’s essentially a lottery. | think 
we’re not willing to accept that so much suc- 
cess in life is just randomness, and in our capital- 
istic economy there’s this constant push for more 
“rationality”, whether or not that makes sense. 


So the way hiring works is a) fundamentally arbi- 


trary because these outcomes are hard to pre- 
dict, and b) there’s a lot of bias along all axes 
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that we know about. What these tools promise 

to do is cut down on bias that is relatively easy 

to statistically quantify, but it’s much harder to 

prove that these tools are actually selecting can- 
didates who will do better than candidates who 

were not selected. The companies who are buy- 
ing these tools are either okay with that or don’t 

want to know. Look at it from their perspective: 

they might have a thousand applications for two 

positions. It’s an enormous investment of time 

to read those applications and interview those 

candidates, and it’s frustrating not to be able 

to make decisions on a clear candidate rank- 
ing. And against this backdrop emerges a tool 

that promises to be Al and has a veneer of sci- 
entific sophistication. It says it will cut down on 

bias and find the best candidates in a way that 

is much cheaper to their company than a tradi- 
tional interview and hiring process. That seems 

like a great deal. 


FK So what you’re saying is the domains in 

which snake oil is more prevalent are the 
ones where either the market is broken or 
where we have a desire for certainty that maybe 
doesn’t exist? 


AN | hesitate to provide some sort of sweep- 

ing characterisation that explains where 
there is a lot of snake oil. My point is more that if 
we look at the individual domains, there seem to 
be some important reasons why there are buyers 
in that domain. We have to look at each specific 
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domeéin and see what is specifically broken there. 

There’s also a lot of Al snake oil that’s being sold 

to governments. | think what’s going on there is 

that there’s not enough expertise in procurement 
_ departments to really make nuanced decisions 
about whether this algorithmic tool can do what 
it claims. 


LDF 0.0576 


FK Do you think this problem is limited to prod- 

ucts and services that are being sold or is 
this also something you observe within the sci- 
entific community? 


LDR 0.0444 


AN A lot of my thinking evolved through 
the “Limits to Prediction” course that | 
co-taught with Professor Matt Salganik, whom 
| mentioned earlier. We wanted to get a better 
scientific understanding of when prediction is 
even possible, and the limits of its accuracy. One 
of the things that stuck out for me is that there’s 
also a lot of misguided research and activity 
around prediction where we have to ask: what 
is even the point? 
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One domain is political prediction. There’s a great 
book by Eitan Hersch which criticises the idea of 
politics, and even political activism, as a sport—a 
horse race that turns into a hobby or entertdin- 

ment. What |! find really compelling about this 
critique is what it implies about efforts like 


FiveThirtyEight that involve a lot of statistics 
and technology for predicting the outcomes of 
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various elections. Why? That’s the big question 
to me. Of course, political candidates themselves 
might want to know where to focus their cam- 
pcigning efforts. Political scientists might want 
to understand what drives people to vote—those 
are all great. But why as members of the public...? 


LDR 0.0408 


Let me turn this inwards. I’m one of those people 

who refreshes the New York Times needle and 

FiveThirtyEight’s predictions. Why dol participate 

in this way? | was forced to turn that critique on 

myself, and I realised it’s because uncertainty 
is so uncomfortable. Anything that promises to 

quell the terror that uncertainty produces and 

tell us that “there’s an 84% chance this candidate 

will win” just fills a huge gap in our subconscious 

vulnerabilities. | think this is a real problem. It’s 

not just FiveThirtyEight. There’s a whole field of 
research to figure out how to predict elections. 
Why? The answer is not clear at all. So, it’s not 

just in the commercial sphere, there’s also a lot 

of other misguided activity around prediction. 
We’ve heard a lot about how these predictions 

have not been very successful, but we’ve heard 

less about why people are doing these predic- 
tions at all. 
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FK Words like “~pseudoscience” and “snake oil” 
are often thrown around to denote anything 
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from harmful Al, to poorly-done research, to 
scams, essentially. But you chose your words 
very carefully. Why “misguided research” rather 
than, let’s say, “pseudoscience”? 
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AN I think all these terms are distinct, at least 
somewhat. Snake oil describes commer- 
cial products that are sold as something that’s 
going to solve a problem. Pseudoscience is 
where scientific claims are being made, but 
they’re based on fundamentally shaky assump- 
tions. The classic example is, of course, a paper 
on supposedly predicting criminality from facial 
images. When I say “misguided research”, a 
good example is electoral prediction by political 
scientists. This is very, very careful research con- 
ducted by very rigorous researchers. They know 
their statistics, | don’t think they’re engaged in 
pseudoscience. By “misguided” | mean they’re 
not asking the question of “who is this research 
helping?” 


















FK That’s really interesting. The question you’re 

asking then is epistemological. Why do you 
think this is the case and what do you see as 
the problems arising from not asking these 
questions? 


AN That’s a different kind of critique. It’s 

not the same level of irresponsibility as 
some of this harmful Al present in academia 
and on the street. Once an academic com- 
munity decides something is an important 
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research direction, then you stop asking the 
questions. It’s frankly difficult to ask that 
question for every paper that you write. But 
sometimes an entire community starts down 
a path that ultimately leads nowhere and is 
not going to help anybody. It might even have 
some harmful side-effects. There’s interesting 
research coming out that the false confidence 
that people get from seeing these probabil- 
ity scores actually depresses turnouts. This 
might be a weird thing to say right after an 
election that saw record levels of turnout, 
but we don’t know whether even more peo- 
ple might have voted had it not been for this 
entire industry of predicting elections, and 
splashing those predictions on the front- 
pages. This is why misguided research is, | 
think, a separate critique. 


FK Moving onto a different theme, | have 
two questions on the limit of predicta- 
bility. It seems like every other year a research 
paper tries to predict criminality. The other 
one for me that surprisingly doesn’t die is 
a 2017 study by two Stanford researchers 
on predicting homosexuality from faces. 
There are many, many problems with this 
paper, but what still fascinates me is that 
the conversations with policymakers and 
journalists often revolved around “Well 
maybe we can’t predict this now, but who 


















knows if we will be able to predict it in 
future@”. In your talk you said that this is an 
incomplete categorisation of tasks that Al 
can be used to solve—and | immediately 
thought of predicting identity. It’s futile, 
but the reason why ultimately lies some- 
where else. It’s more a question of who we 
think has the ultimate authority about 
who defines who we are. It’s an ontological 
question rather than one about accuracy 
or biology. | am curious how you refute 
this claim that /\ will be able to predict 
things in the future, and place an inherent 
limit on what can be predicted? 
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AN If we 


look at 
the authors of the paper on predicting 
sexual orientation, one of their mein sup- 
q posed justifications for writing the paper is 
they claim to be doing this in the interest 
of the gay community. As repressive gov- 
ernments want to identify sexuality through 
photos and social media to come after peo- 
ple, they think it’s better for this research to 
be out there for everybody to see and take 
defensive measures. 


é 


E 10700 


32 Al SNAKE OIL, PSEUD 


LDF 0.0291 





90 
a 
Ss 
06 
ie) 
4 


E 11200 


| think that argument makes sense in some 
domains like computer security. It absolutely 
does not make sense in this domain. Doing this 
research is exactly the kind of activity that gives 
a veneer of legitimacy to an oppressive govern- 
ment who says “Look! There’s a peer-reviewed 
research paper and it says that this is scientifi- 
cally accurate, and so we’re doing something 
that’s backed 
by science!” 
Papers like this 
give ammunition 
to people who might do 
such things for repressive 
ends. The other part is that if you find a vulnera- 
bility in a computer program, it’s very easy to fix— 
finding the vulnerability is the hard part. It’s very 
different in this case. If it is true (and of course 
it’s very doubtful) that it’s possible to accurately 
infer sexual orientation from people’s images on 
social media, what are these authors suggesting 
people do to protect themselves from oppres- 
sive governments other than disappear from 
the internet? 





FK I think that the suggestion was “accept the 
death of privacy as a fact and adapt to social 
norms” which... yeah... 


AN Right. | would find the motivations for 

doing this research in the first place to be 
very questionable. Similarly, predicting gender. 
One of the main applications is to put a camera 
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in the back of the taxi that can infer the 

rider’s gender and show targeted advertise- 
ments on the little television screen. That’s 

one of the main applications that I’m seeing 

commercially. Why? You know... | think we 

should push back on that application in the 

first place. And if none of these applications 

make sense, we should ask why people are 

even working on predicting gender from 

facial images. 
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FK So you would rephrase the question 
and not even engage in discussions 

about accuracy, and just ask whether we 

should be doing this in the first place? 


AN That’s right. | think there are several 

kinds of critique for questionable uses 
of 4). There’s the bias critique, the accu- 
racy critique, and the questionable appli- 
cation technique. | think these critiques 
are separate (there’s often a tendency to 
confuse them) and what | tried to doin the 
i! Snake Oil talk is focus on one particular 
critique, the critique of accuracy. But that’s 
not necessarily the most relevant critique 
in all cases. 
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FK Let’s talk about Al and the current 
state of the world. | was moderately 
optimistic that 

there was less Al 

: solutionism in 
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response to Covid-19 
than I feared. Could 
this be a positive indi- 
cator that the debate has 
matured in the past two years? 


AN It’s hard to tell, but that’s a great question. 

It’s true that companies didn’t immedi- 
ately start blowing the Al horn when Covid-19 
happened, and that is good news. But it’s hard to 
tell if that’s because they just didn’t see enough 
commercial opportunity there or because the 
debate has in fact matured. 


FK There could be various explanations 
for that... 


AN Yeah. There is a lot of snake oil and mis- 
guided AJ in the medical domain. You 
see a lot where machine learning was tested 
on what is called a “retrospective test”, where 
you collect data first from a clinical setting, 
develop your algorithm on that data and then 
just test the algorithm on a different portion of 
the same data. That is a very misleading type 
of test, because the data might have been col- 
lected from one hospital but when you test it 
on a different hospital in a different region— 
with different cultural assumptions, different 
demographics—where the patterns are differ- 
ent, the tool totally fails. We have papers that 
look at what happens if you test these retro- 
spectively-developed tools in a prospective 
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clinical setting: there’s a massive gap in accura- 
cies. We know there’s a lot of this going on in the 

medical machine learning dom2in, but whether 

the relative dearth of snake oil Al for Covid-19 is 

due to the debate maturing or some other fac- 
tor, who can tell. 


LDF 0.0368 


FK One thing | was wondering... do you feel like 
you’ve made an impact? 


AN (laughs) 


FK Asin, are you seeing less snake oil now than 
you did, say two years ago? 
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AN That’s hard to know. | think there is cer- 
tainly more awareness among the people 
who’ve been doing critical_,Al work. I’m see- 
ing less evidence that awareness is coming 
through in journalism, although I’m optimis- 
tic that that will change. | have a cou- 
ple of wish-list items for journalists 
who often unwittingly provide 
cover for overhyped cltcms. 
One is: please stop attributing 
agency to /s. | don’t understand 
why journalists do this (presuma- 
bly it drives clicks?) but it’s such a 
blatantly irresponsible thing to do. 
Headlines like “AT discovered how 
to cure a type of cancer”. Of course 
it’s never M that did this. 
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It’s researchers, very hardworking re- 
searchers, who use A! machine learning 
tools like any other tool. It’s both demean- 

ing to the researchers who did that work 
and creates massive confusion among the public 
when journalists attribute agency to A?. There’s 
no reason to do that, especially in headlines. 


LDF 0.069 


And number two is that it’s virtually never mean- 
ingful to provide an accuracy number, like “A 

used to predict earthquakes is 93% accurate”. | 

see that all the time. It never makes sense in a 

headline and most of the time never makes sense 

even in the body of the article. Here’s why: | can 

take any classifier and make it have just about 

any accuracy | want by changing the data distri- 
bution on which | do the test. | can give it arbi- 
trarily easy instances to classify, | can give it 

arbitrarily hard instances to classify. That choice 

is completely up to the researcher or the com- 
pany that’s doing the test. In most cases there’s 

not an agreed-upon standard, so unless you’re 

reporting accuracies on 

a widely-used, agreed- 

upon benchmark 

dataset (which is 

virtually never the 

case, it’s usually the company deciding on-the- 
road how to do the test) it never makes sense 

to report an accuracy number like that without 

a lengthy explanation and many, many other 
caveats. So don’t provide these oversimplified 

headline accuracy numbers. Try to provide these 
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caveats and give qualitative descriptions 
of accuracy. What does this mean? What 
are the implications if you were to employ 
this in a commercial application? How often 
would you have false positives? Those are 
the kinds of questions that policymakers 
should know, not these oversimplified 


accuracy numbers. 
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1. This definition was partly inspired by Rabinowitz, A. (2021, January 8) 

Cheap talk skepticism: why we need to push back against those who are 
‘just asking questions’. The Skeptic. https://www.skeptic.org.uk/2021/01/ 
cheap-talk-skepticism-why-we-need-to-push-back-against-those-who- 
are-just-asking-questions 


Cheap talk (n): 

talk that results in 
harm to the marginal- 
ised but costs nothing 
for the speaker: ' 


Not a week goes by 

without the publi- 

cation of an acade- 

mic paper or AT tool claiming to predict 
gender, criminality, emotion, personal- 


ity, political orientation or another social 
attribute using machine learning. Critics 
often label such work as pseudoscience, 
digital phrenology, physiognomy, Al snake 
oil, junk science, and bogus A\. These la- 
bels are fitting and valid. However, iden- 
tifying this work as Cheap captures the 
fact that those producing it (usually a ho- 
mogeneous group from privileged back- 
grounds) suffer little to no cost, while the 
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people who serve as the testing grounds, 
frequently those at the margins of soci- 
ety, pay the heaviest price. 
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Cheapness emerges when a system 
makes it easy to talk at little or no cost 
to the speaker, while at the same time 
causing tangible harm to the most vul- 
nerable, disenfranchised, and under- 

served. Within traditional sciences, 

cheapness manifests when racist, 
sexist, ableist, misogynist, trans- 
phobic and generally bigoted as- 
sumptions are re-packaged as 
scientific hypotheses, with the 
implication that the only viable 
way to reject them is to test them. | 

Much of the work from the inti- | 
mately related fields of “race sci-_ 

ence” and IQ research constitutes 

Cheap science. Increasingly, parts 

of Al research and its applications 
are following suit. 


Cheap Al, a subset of Cheap 
science, is produced when AT is 
inappropriately seen as a solu- 

tion for challenges that it is not 
able to solve. It is rooted in the 
faulty assumption that qualities 
such as trustworthiness, emotional 
state, and sexual preference are 
static characteristics with physical 








expression that can be read (for example) from 
our faces and bodies. Software that claims to 
detect dishonesty, most facial recognition sys- 
tems deployed in the public sphere, emotion 
and gait recognition systems, AJ that catego- 
rises faces as “less or more trustworthy”—all of 
these constitute Cheap Al. Judgements made 
by these systems are inherently value-laden, 
wholly misguided and fundamentally rooted in 
pseudoscience. 
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At the root of 
Cheap Al are preju- 
2. Saini, A. (2019) Superior: the diced assumptions 
return of race science. Boston, masquerading as objec- 
piass:Beacon ag tive enquiry. For instance, 
the very conception of racial 
categories derives from German 
doctor Johann Friedrich Blumenbach’s 
declaration in 1795 that there are “five 
human varieties: Caucasians, Mongolians, 
Ethiopians, Americans, and Malays”.’ This 
arbitrary classification placed white people 
of European descent at the top of the hier- 
archy and cleared the way for colonisation. 


CHEAP Al 





Subsequently, appar- 
ently scientific racial 
classifications have 
served as justifica- 
tions for inhu- 
mane actions that 
~ include naturalised 
slavery, forced ster- 
- ilization, the genocidal Nazi 
attempt to exterminate the 
Jewish people, people with 
disabilities, and LGBTQ+ peo- 
ple; among other “deviant” 
classes. Today, racial classifi- 
cations continue to justify dis- 
criminatory practices—inclu- 
ding immigration policy—as a 
way to filter out “inferior” races. 









Like other pseudosciences, 
Cheap science is built around 
oversimplifications and misinter- 
pretations. The underlying objec- 
tive of “race science”, for example, 
[ is to get at biological or cogni- 
tive differences in supposed ca- 
pabilities between different races, 
ethnicities, and genders, on the 
presumption that there exists a hi- 
erarchy of inherent differences to be 
found between groups. IQ research, 
for instance, has asserted the ex- 
istence of race-based differences, 
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by reducing intelligence to a 

single number and by framing it as dependent 
upon race. Similarly, Cheap Al arrogates com- 
plex and contingent human behaviour to a face, 
gait, or body language. Tools based on these 
presumptions are then produced en masse to 
classify, sort, and “predict” human behaviour 
and action. 


Cheap A’ presents itself as something that ought 
to be tested, validated, or refuted in the mar- 
ketplace of ideas. This grants bigoted claims the 
status of scientific hypothesis, and frames the 
proponents and critics of Cheap Al as two sides 
of equal merit, with equally valid intent and equal 
power. This equivalence is false. While those cre- 
ating or propagating Cheap I may face criticism, 
or reputational harm (if they face these things 
at all), marginalised people risk discrimination, 
inhumane treatment, or even death as a result. 


Time and time again, attempts to find meaning- 
ful biological differences between racial groups 
have been proven futile, laden with error (there 
exist more average differences within groups 
than between groups), and rooted in racist moti- 
vations. Yet, the same speculations persist today, 
just differently framed. In a shift precipitated 
by the catastrophic effects of Nazi “race sci- 
ence”, race and IQ research has abandoned the 
outright racist, colonialist, and white suprem- 
acist framings of the past, and now masquer- 
ades in cunning language such as “populations”, 
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“human variation”, and “human biodiver- 
sity” research. 
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The mass application of Cheap Alhas | 
had a gradual but calamitous effect, es- | 
pecially on individuals and communi- _ | 
ties that are underserved, marginalised, 
and disproportionately targeted by these 
systems. Despite decades of work 
warning against the dangers of a 
reductionist approach, so-called 
“emotion detection systems” 
continue to spread. Though 
criminality is a largely complex 
social phenomenon, claims are 
still made that Al systems can de- 
tect it based on images of faces. 
Although lies and deception are 
complex behaviours that defy 
quantification and measurement, 
assertions are still made that they 
can be identified from analysis 
of video-feeds of gaits and ges- 
tures. Alarmingly, this and simi- 
lar work is fast becoming mcin- 
stream, increasingly appearing 
in prestigious academic venues 
and journals such as NeurIPS and 
Springer. 
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The forms Cheap Al takes, 
the types of claims made of 
it, and the areas to which it 


is applied, are varied and fast expanding. 
Yet, a single theme persists: the least priv- 
ileged, the most disenfranchised, and the 
most marginalised individuals and commu- 
nities pay the highest price. Black men are 
wrongly detained due to failures in facial 
recognition systems; Black people with _ , 
ill health are systematically excluded 
from medical treatment; the elderly and | 
women are not shown job ads. These few 
cases represent only the tip of the ice- 
berg. So far there are three known cases of 
Black men wrongly detained due to facial 
recognition; there are likely many more. The 
victims of algorithmic injustice that we know 
about are often disenfranchised, and it is 
likely that many more are fighting injustice 
in the dark, or falling victim without redress, 
unaware that Cheap 4’ is responsible. 





Like segregation, much of Cheap Al is built on 
a logic of punishment. These systems embed 
and perpetuate stereotypes. From “decep- 
tion detection” to “emotion recognition” sys- 
tems, Cheap 4! serves as a tool that “catches” 

and punishes those deemed to be outliers, 
problematic, or otherwise unconventional. 


The seemingly logical and open-minded 
course of action—to withhold judgement 
on these systems on the premise that their 
merits lie in how “well” or “accurately” they 
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work—lends them a false sense of reasonable- 
ness, giving the impression that the “self-cor- 
recting” nature of science will eliminate bad 
tools. It also creates the illusion of there being 
“two sides”. In reality, criticisms, objections, 
and calls for accountability drown in the sea 
of Cheap [| that is flooding day-to-day life. 
Cheap Al is produced at an unprecedented rate 
and huge amounts of money go into producing 
it. By contrast, those working to reveal it as sci- 
entifically unfounded and ethically dangerous 
are scholars and activists working under precar- 
ious positions with little to no support, who are 
likely to suffer negative consequences. 


By suspending judgement until wrong is proved, 
an ecosystem has been created where anyone 

can claim to have created obviously absurd and 

impossible tools (some of which are nonetheless 

taken up and applied) without facing any conse- 
quences for engaging in Cheap 4!. Such creators 

and deployers may risk their reputations when 

their tech is proven to be “inaccurate”. However, 
for those who face the burn of being measured 

by this tech, it can be a matter of life and death, 
resulting in years lost trying to prove innocence, 
and other grave forms of suffering. 


There is no quick solution 
» to ending Cheap Ml. Many 
: factors contribute 
™ to the ecology in 
which it thrives. 
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This includes blind faith in &, the illusion of 
objectivity that comes with the field’s asso- 
ciation with mathematics, Cheap /I!’s creators 
and deployers’ limited knowledge of history 
and other relevant fields, a lack of diversity and 
inclusion, the privilege hazard (a field run by 
a group of mostly white, privileged men who 
are unaffected by Cheap AMi’s harms), the ten- 
dency to ignore and dismiss critical voices, and 
a lack of accountability. We must recognise 
Cheap AI as a problem in the ecosystem. All 
of these factors and more need to be recog- 
nised and challenged so that Cheap, Al is seen 
for what it is, and those producing it are held 
~ accountable. 
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It took Nazi-era atrocities, forced steriliza- 
tions, and other inhumane tortures for phre- 
nology, eugenics, and other pseudosciences 
| toberelegated from science’s mainstream to 
its fringe. It should not take mass injustice for 
|_| Cheap AI to be recognised as similarly harm- 
| ful. In addition to strict legal regulation and 
the enforcement of academic standards, we 
ourselves also bear a responsibility to 
call out and denounce Cheap /s, and 
those who produce it. 
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1. Charette, R. N. (2018, 24 January) Michigan’s MiDAS Unemployment 
System: Algorithm Alchemy Created Lead, Not Gold, IEEE Spectrum. 
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On August 
29th 1907, the Siskse Bridge col- 
lapsed. Bound for the record books 
as the longest cantilever bridge ever 
to be built, the disaster cost 75 of the 
86 lives working on its construction 
that day. In the formal inquiry that fol- 
lowed, the cause of the accident was 
traced to the poor oversight of the 
two lead engineers, Theodore Cooper 
and Peter Szlapka. The landmark dec- 
laration on engineering responsibility 
called out both by name. Their avoid- 
able missteps, careless miscalcula- 
tions, and chronic disregard for safety 
were described in detail and ultimately 
identified as key to the bridge’s struc- 
tural failure. 


Decades later, in 1965, US consumer advocate 
Ralph Nader published Unsafe at Any Speed, a 
book which would transform the automotive 
industry. In it, Nader revealed the alarming 
instability of the Chevrolet Corvair—a vehi- « 
cle which, for him, exemplified the auto | | 
industry’s indifference towards safety. 

He noted an absence of foresight in how 
cars were designed and actually used, and 
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connected this to a string of avoidable col- 
lisions and accidents. He flagged 
this as a new era, one marked 
by its own kind of collapse. 
In lieu of careful planning 
and precaution, he 
observed instead only 
“an explosion of what is 
actually done”. 

















2. Obermeyer, Z., Powers, B., 
a P a P Vogeli, C. & Mullainathan, S. (2019) 
Such explosions persist Dissecting racial bias in an algorithm 


: : H used to manage the health of populations. 
today bie the So esr Science 366.6464: 447-453. DOI: 
of algorithmic systems. 10.1126/science.aax2342 
The rubble is everywhere. 3. Raji, 1. D. & Buolamwini, J. (2019) 
The deployment of the Actionable auditing: Investigating 
MIDAS alaorithm led to the impact of publicly naming biased 

g a performance results of commercial AI 
over 20,000 Michiga- products. Proceedings of the 2019 AAAI/ 
nians being erroneousl ACM Conference on Al, Ethics, and 
a = Society. DOI: 10.1145/3306618.3314244 

accused of unemploy- 

1 4. Lee, K. (2019, 6 November) Uber’s 
ment fraud. A model . Self-Driving Cars Made It Through 
widely used in US hospi- 37 Crashes Before Killing Someone. 
tals to allocate health- Jalopnik. https://jalopnik.com/ubers-self- 

7 . driving-cars-made-it-through-37-crashes- 
care systematically dis- WReCsrea 
criminates against Black 
patients.’ Several facial 
recognition products fail 
on Black female faces.’ 
Uber’s self-driving car 
software has led to doz- 
ens of accidents and even 
a human death.’ In each 
case, the investigated 


causes are increasingly 
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inexcusable—unin- 
spected “corrupt and 
missing” data, the 

admittedly incorrect 
framing of a task, a re- 
markable lack of over- 
sight in testing, or an 5. Ledford, H. (2019) 

unimplemented feature. IY CODE Toyetsaroy io) F-Te) <q vero} 0) (e 


affected by racial bias 
in health-care algorithms. 


The harm inflicted by RRA 
these products is adirect "QqRaERaire 
consequence of shoddy mee28-6 
craftsmanship, unacknowl- 

edged technical limits, and poor 
design—bridges that don’t hold, 

cars that can’t steer. Contrary to popular belief, 
these systems are far more likely to cause harm 
when they do not work than on the rare occa- 
sion in which they work “too well”, or in truly 
unaccountable ways. 






Even in light of these obvious harms, vendors 
remain stubborn—and defensive. Despite a 93% 
error rate, the MiDAS algorithm was used for at 
least three years. It took several highly publi- 
cised lawsuits for its use to finally be questioned. 
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= Optum, the company behind the healthcare pri- 

=  oritisation algorithm, called audit results helpful 

A but “misleading” and continues to deploy the 
system on millions of patients.° In spite of a peer 
reviewed study revealing an over 30% error rate 
on darker female faces, Amazon still pene 
its biased facial recognition = 
technology for nation-wide 

<= partnerships with law enfor- 

= cement. Uber’s automated | 

A vehicles, which are still being tested on public 
streets, continue to run on its flawed software. 
Just as the car manufacturer called out by Nader 
shifted blame onto car dealerships for failing 
to recommend tyre pressures 
to “correct” the Corvair’s 
faulty steering, algorithm 
developers also seek 
scapegoats for their 

~ own embarrassing fail- 

& ures. Optum states that 

Q the quality of its healthcare 


algorithm’s assessments 
actually depends on “the 
doctor’s expertise”. Amazon 
discusses setting higher accu- 
racy thresholds for police clients, 
and MiDAS’s creators point to the 
difficulty of migrating from an in- 
dependent legacy system. Even the 
fatal Uber self-driving car crash was ul- 
timately blamed on a distracted test driver 
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and the victim’s own jaywalking. Those 
creating these algorithms will point to 
anything—the human operator, institu- 
tional infrastructure, society-at-large, 
or even the actions of the affected pop- 

ulation itself—before admitting to in- 

herent flaws in the product’s design 

and implementation. Eyes averted, they 
claim the issues reported are really some- 
one else’s responsibility. 


However, just as it was General Motors’ 
responsibility to ensure that the 
Corvair’s required tyre pressures 
were within the range of recommen- 
ded tolerance, it is also Amazon’s 
responsibility to inform police 
clients to operate at a level other 
than the default accuracy threshold. 
It is Uber’s responsibility to ensure drivers 
remain adequately attentive at the wheel, 
the MiDAS developers’ task to make their 
algorithm portable, and Optum’s role to an- 
alyse the causal assumptions in their data. 
To forget a test, neglect design or dismiss the 
data is not just unfortunate, it’s irresponsible. 


Any institutional stakeholder involved in the 
development of a product—be they engineer, 
executive business authority, or product or 
marketing participant—does, ultimately, 
have an impact on outcomes. The tech- 
nical community tends to simultaneously 
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anthropomorphise AI systems, ascrib- 
ing to them a false ability to “think” and 
“learn” independently of the provided input, 
while also downplaying, with language of data 
“bricks”, “moats” and “streams”, the existence 
of real humans in the data and human participa- 
tion in the decisions that shape these systems. 
This reluctance to admit to the human influ- 
ence on A] functionality makes the field stub- 
bornly blind to its contribution to these systems. 
Technologists are not like doctors, looking each 
patient in the eye. They stand at a distance, the 
relationship between their judgement and sys- 
tem outcomes blurred by digitised abstraction, 
their sense of responsibility dampened by scale, 
the rush of agile innovation, countless undocu- 
mented judgements, and implicit feature engi- 
neering. The result is an imagined absolution of 
responsibility, a false narrative in which they’ve 
created an artificial system outside of anyone’s 
control, while the human population affected 
by their decisions and mistakes is inappropri- 
ately erased. 


As a former participant in an Applied Machine 
Learning team, I’ve witnessed the model devel- 
opment process up close. It’s a chaotic enter- 
prise. Whether building a moderation model that 
disproportionately 
filtered out the 
content of 
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people of colour, or training a heir classifier that 
did not have inclusive categories, we regu- 
larly disappointed our clients and ourselves, 
frequently falling short of expectations in 
practice. Even after a live pilot failed or 
unacceptable bias was discovered, at the 
client’s request we would deploy the model 
anyway. The fact is, in this field, inattentive 
engineers often get away with it. No data 
review is imposed, nor reporting require- 
ments. There is no sanctioned communi- 
cation protocol with operators, no safety 
culture to speak of, no best practices to 
adhere to, no restrictive regulations or , 
enforced compliance, and no guide 
for recall—voluntary or imposed—to 
remove from our daily lives the mod- 
els that cannot be steered, the algo- 
rithms without brakes. 


Of the 75 construction 
workers who died in the é 
Quebec Bridge catastro- 
phe, up to 35 were Native 
Americans, Mohawks from the 
Kahnawake community who faced a dearth of 
employment options at the time and received 
poverty wages. This is what happens when sys- 
tems collapse: they fall on the most vulnerable. 
In his book, Nader described Ms. Rose Pierini 
“learning to adjust to the loss of her left arm” 
after a crash caused by the unaddressed 
steering challenges of the Chevrolet Corvair. 
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He profiled Robert Comstock, a “veteran 
garage mechanic”, whose leg was ampu- 
tated after he was run over by a Buick with no 
brakes. Today, we discuss Robert Williams, © 
a Black man wrongfully arrested due to 
an incorrect facial recognition “match”; | 
Carmelita Colvin, a Black woman falsely 
accused of unemployment fraud; Tammy | 
Dobbs, a sickly older lady who lost her 
healthcare due to a program glitch, and | 
Davone Jackson, locked out of the low-in- | 
come housing he and his family needed to 
escape homelessness due to a false report 
| from an automated tenant screening tool. | 
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. The fact is, there are bodies underneath | 
' the rubble. The individuals caught in the 
web of .{!’s false promises are just the lat- 
est casualties of technological collapse. It 
. willnever be enough to feign ignorance, 
defiect blame, and keep going. For the 
sake of those that suffer, responsibility is 
required. It’s a concurrent goal, of a dif- 
fering urgency, to address alongside any 
) ambition for a more hopeful future. 


Deborah Raji is a Mozilla fellow, interested 
in algorithmic auditing. She also works 
closely with the Algorithmic Justice | 
League initiative to highlight bias 
in deployed products. 
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Who am | as data? The question has haunted 
me for years. It’s a ghoulish curiosity; wanting 
to know not just how much data companies 
routinely harvest about my behaviour, but also 
what this data might reveal about me, and who 
| appear to be. 


It also stems from my fascination with a par- 
adoxical societal shift. We are living at a time 
when the terms we use to describe identities 
are becoming more fluid, and boundaries more 
negotiable. While our language falls short of 
fully representing our social realities, it reflects 
the fundamental changes that are afoot—pro- 
viding an echo of what is already the case. At 
the same time, another current is pulling in 
the opposite direction. Out of sight, we are 
increasingly surrounded by data-driven sys- 
tems that automatically affix names to us and 
assign us an identity. From the gender binary 
that’s encoded into tar- 
_ geted online advertising, 
to facial recognition sys- 
tems that claim to predict 
people’s gender or eth- 
nicity, these systems are 
often completely inade- 
1. Pee VMS) Laskee quate to our social 
an online tracking company for all 2 
of my data and here’s what I found. realities. Yet 
ee simply by ex- 
isting, they classify and 


read/2433/i-asked-online- 


tracking-company-all-my- 
data-and-heres-what-i-found thereby mould the world 
) around us. 
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In 2018, | asked an advertising company for all of 
my' data. Quantcast, an Al company that is known 
for its cookie consent notices, tracks users across 
millions of websites and apps to create detailed 
profiles of people’s browsing histories, presumed 
identity, and predicted interests. 


Staring into the abyss of one’s tracking data is 
uncanny. On the one hand, there were pages and 
pages of my own browsing history—not just top- 
level domains, but URLs that revealed exactly 
what | had read and clicked on, which restau- 
rants I’d booked, and which words I'd translated 
into German. Then there were the predictions 
made about me, confidence intervals about my 
predicted gender, age, and income. Had | done 
the same experiment outside the European Un- 
ion—where this is unlawful—the data would have 
included an ethnicity score. 


Was this who | am? | expected to feel somewhat 
violated, but not so misunderstood. These rows 
and rows of data revealed a lot about me, my in- 
terests, and how | spend my time (every click is 
timestamped and carries other metadata) but 
the picture remained incomplete, shallow. Eerily 
accurate inferences were combined with seem- 
ingly random consumer categories: bagel shop 
frequenter, alcohol consumption at home. 


Al-driven profiling for advertising may sound ba- 
nal, yet it is the most profitable and prevalent 





use of Al to classify, infer, and detect people’s 
identity—be it gender, ethnicity or sexual ori- 
entation. These Al systems operate in the back- 
ground, in ways that are fundamentally beyond 
people’s knowledge or control. Those who are 
classified and assessed by them frequently don’t 
know where, when or how these systems 

are used. These systems are deeply Orwel- 

lian when they happen to get it right, and 
Kafkaesque when they do not. 


Just to be crystal clear: it is impossible to detect 
someone’s gender, ethnicity, or sexual orienta- 
tion using Al techniques, and any attempt to do 
so falls squarely in the domain of Al pseudosci- 
ence. That is not because Al isn’t good enough 
yet, or because we need more and better data. 
It’s because identity isn’t something that can be 
straightforwardly detected, like the colour or 
shape of an object. 


Identity is a fallacy, argues Kwame Anthony 
Appiah in Rethinking Identity’. The fallacy is in 
_ assuming that there is some deep 

= similarity that binds together 
people of a particular iden- 
. tity—because ultimately 
such a similarity “ does not exist—and yet, 
identities matter deeply to people pre- 
cisely because belonging to something larger 
than oneself is a key human need. To acknowl- 
edge the ways identity is at once true and a lie 
reveals the tension at its heart. The negotiations 
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between how we identify ourselves and how oth- 
ers identify us, what we are identifying with and 
what we’re being identified as, are perpetual. 


A\-driven identity prediction attempts to 
dissolve this tension by reducing com- 
plex questions of identity to what is automat- 
ically detectable from the outside, according 
to pre-conceived notions of what an identity is. 
At best, fl systems can offer a very incomplete 
guess of someone’s presumed identity—the da- 
ta-driven and automated equivalent of the ste- 
reotyping that people regularly do to passing 
strangers in the street. At worst, Al systems that 
claim to detect identity give racism, transphobia 
and sexism a veneer of scientific advancement 
and objectivity. This has devastating conse- 
quences for those who are either targeted or 
rendered invisible by these systems. As Sasha 
Costanza-Chock describes in Design Justice, 
the gender binary that is encoded into so much 
infrastructure, such as airport security systems, 
» amounts to ontological re- 
duction: “As a non- 
binary trans , 
femme, 









2. Appiah, K. A. (2018). 
The lies that bind: Rethinking 
identity. London: Profile Books. 
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| present a problem not easily resolved by the 
algorithm of the security protocol.” 


Broadly speaking, there are two ways | 
in which Al is used commercially to as- 
sess, detect or predict identity. Let’s 
call them the pragmatic and the essen- | 
tialist approaches. The former predom- | 
inates in online advertising. Ultimately, 
it doesn’t matter to Quantcast, Google or 
Facebook whether | truly am a woman or a 
man (or identify as trans, an option that is of- 
ten not available). What matters is whether 
my purchasing decisions and online behav- 
iour resemble that of other people who are 
also classified as “women”. In other words, 
“woman.” is not really a statement about 
identity, as John Cheney-Lippold writes in 
We Are Data’. It is an arbitrary label that is 
applied to an arbitrary group of people who 
share an arbitrary set of online behaviours. 
What determines the label and the group is 
not whether itis true, but whether grouping 
people together increases whatever the ad- 
vertising system is optimised for: clicks, time 
on site, purchases. 





By contrast, the essentialist approach posi- 
tions automated predictions about identity 
as truth claims. This is where pseudoscience 
comes in. The idea behind security cameras 
that detect Uyghur ethnicity is to really de- 
tect and spot people of Uyghur descent in 
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3. Costanza-Chock, S. (2018). a crowd. When a wide- 
Pg USMS SS Was ly-criticised and now-in- 
an intersectional feminist 
framework for design theory famous Stanford study 
and practice. Proceedings of claimed that Al can de- 
the Design Research Society. 
tect whether someone is 
EE REMI 927 oF straight based only 
New York University Press. ona photo, the under- 
lying premise was that 
sexual orientation is 
really somehow 
visually inscribed 
in a face. And when 
immigration author- 
ities use voice rec- 
ognition to detect 
accents to deter- 
mine nationality, 
_ the unspoken as- 
sumptionis “that nationality, 
which is primarily a matter of legal ju- 
risdiction, is expressed in and deter- 
mined by an accent. These systems are 
designed to detect deviations from the 
mean, and thereby they also inevitably 
define what is considered ‘normal’ and 
‘not-normal’, where anything that isn’t 
‘normal’ is automatically suspicious. 







Both approaches are problematic. The 

pragmatic (or rather, “just good enough”) 
approach to profiling may still lead to discrimi- 
nation, because predictions can be both eerily 
accurate and wholly inaccurate or inadequate 
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} at the same time. Whether accurate or not, 
* inferring someone’s ethnicity or gender with- 
- out their knowledge or consent still means 
this “information” can be used in ways which 
discriminate or exclude. Inferences that are 
produced for contexts which make do with a 
pragmatic approach—for example, in adver- 
tising—may still end up in a context af 

where accuracy , ae 
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and 
truth matter. 
It’s one thing to target an ad based on some- 
one’s likely interests, whereabouts and ethnic- 
ity; it’s something entirely different to use the 
same data for immigration enforcement. 


















What’s so deeply troubling about the essential- 
ist approach to ‘I identity prediction, by con- /) 
trast, is not just how it operates in practice. | 
These predictions come with significant mar- — 
gins of error, completely eliminating those 
who don’t fit into whatever arbitrary classi- 
fications the designers of the system have 
chosen, and their harms disproportionately 
affect communities that are already mar- 
ginalised. Equally disturbing is the insidious 
idea that automated classification systems 
can, and should be the ultimate authority over 
who we are understood to be. Decisions of this 
magnitude can affect, curtail and even elimi- 
nate the ability to enjoy rights as an individual, 
a citizen, or member of any other group. Yet 
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these systems are intended to be, and are of- 
ten treated as, more reliable, more objective and 
more trustworthy than the statements made by 
those they are assessing. 


This is as troubling as it is absurd, for the essen- 
tialist approach ignores the fact that the under- 
lying and unspoken assumptions, norms, and 
values behind every identity prediction system 
are categories and modes of classification that 
were chosen by the designers of those very sys- 
tems. Take, for instance, the gay faces study 
mentioned earlier. The entire model was based on 
the assumption that people are either gay or 
strcight, male or female. Such binary 
labels fail to capture the lived experi- 
ence of a vast number of queer people— 
not to mention that the study only included 
white people. 


Classifications have consequences, and pro- 
duce real-world effects. In the words of Geof- 
frey Bowker and Susan Star, 
“For any. individual group il 5. Bowker, G. C., & Star, 
situation, classifications and S. L. (2000). Sorting things 
standards give advantage or out: Classification and its 

? . : consequences. Cambridge, 
they give suffering.”° The idea Mvensvinpacs 
that an automated system can 
detect identity risks transfor- 
ming inherently political deci- 
sions—such as opting to use the 
gender binary—into a hard, yet 
invisible infrastructure. Such is 


the case when any form of attribute rec- 
ognition is added into computer vision sys- 
tems, such as face recognition cameras. 





At pseudoscience does not happen in a vac- 
uum. It is part of a much broader revival of 
certain ways of thinking. Amid the return of 
race science and the emergence of DNA tests 
that designate people’s ancestry as “35 per- 
cent German”, or “76 percent Finnish”, the 
use ofl to predict and detect identity needs 
to be seen as part of a much wider revival 
of (biological) essentialism and determin- 
ism’. Many companies sf 
that offer genetic pre- 


dictions, for instance, | pecs, 


sell much more than Superior: The return of 
DNA tests’ They are race science. Boston, 


Mass: Beacon Press. 


also benefiting from— 
and ultimately spread- iiauaaeeMeremes 
ing—the dangerous, yet [iimeaerruen 
incredibly compelling, eas 
idea that who we are is Silla orgiensblog/ 
ultimately determined by Wiaiiiaeeiasiens 
biology. There are now DNA 
companies that claim genet- 
ics can predict people’s ideal 
lifestyle, their intelligence, 
their personality and even 
their perfect romantic part- 
ner. A German start-up makes 
muesli based on people’s DNA, 
house-sharing platform 
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SpareRoom trailed 
genetically matched 
roommates, and the 
music streaming service 
Spotify offers a playlist 

tailored to your DNA. 








At the core of this return to biological 
determinism lies the idea that both peo- 
ple and categories are fixed, unchangea- 
ble and therefore predictable. The dark 
shadow this casts — is the belief that some 
categories of identity, and by extension some 
people’s lives, are superior to others. &merely 
gives outmoded ways of thinking the veneer of 
objectivity and futurism. 


In reality, categories like race, gender, and sex- 
ual orientation evolve over time. They remzin 
subject to contestation. The idea of a criminal 
face is absurd, not least because our ideas of 
criminality are constantly changing. Does our 
face change whenever laws change, or when 
we move to a new country? What are DNA com- 
panies referring to when saying that someone 
is German? The Federal Republic of Germany? 
The Holy Roman Empire of the German Nation 
that existed from 1512 to 1806? Nazi Germany? 
The very idea of the nation state is a modern 
concept. And is someone who recently immi- 
grated to Germany and has a German passport 
not German? 


On the individual level, we are often different 
things to different people. We reveal different 
parts of ourselves in different settings. How we 
see ourselves can evolve or even radically change. 
This freedom to selectively disclose and manage 
who we are to whom, and the space we have to 
do it in, is drastically eroded by the increased 
ability of companies and governments to link 
and join previously distinct data points, both 
spatially and temporally, into a distinct, singular 
identity—an apparently definite assessment of 
who each of us is as a person. 
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In 2020, Google announced that it would 
drop gender recognition from its Cloud 
Vision API, which is used by developers to 
analyse what’s in an image, and can iden- 
tify anything from brand logos to faces to 
landmarks. That’s a good first step, but fur- 
ther action needs to be taken in industry 
more widely. For this we need A! regulation 
that does more than simply setting up 
the standards and guidelines that de- 
termine how Al systems can be used. 

What is needed are clear red lines that 

demarcate what Mcannot and should 

not be used for. 
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Resisting and protecting people from /i 
pseudoscience is about far more than making 
A accountable, expl2inable, more transparent, 
and less biased. It is about defending and vigor- 
ously protecting what it means to be a person. 
It is about resisting ontological reduction. It is 
about allowing space for identities to be chal- 
lenged and contested, not cemented by inscru- 
table automated systems. 


Frederike Kaltheuner is a tech policy analyst 
and researcher. She is also the Director of the 
European AI Fund, a philanthropic initiative to 
strengthen civil society in Europe. 
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Many modern A’ systems are designed 
to ingest and analyze massive datasets 
so as to make recommendations, predic- 
tions, or inferences on unseen inputs 
ranging from images to pieces of text 
and other forms of data. These data- 
sets often reflect patterns of inequity that 
exist in the world,! and yet the data-driven 
nature of /\| systems often serves to obscure 
the technology’s limitations within a perva- 
sive rhetoric of objectivity.” As Al technologies 
and methods are increasingly incorporated 
into all aspects of social life, often in ways that 
increase and accelerate existing social inequi- 
ties, this is especially important. In this piece, 
we examine how and why appeals to objectivity 
are so deeply embedded in technological dis- 
courses and practices. As Ruha Benjamin notes, 
routing algorithmic bias through a rhetoric of 
objectivity can make it “even more difficult to 
challenge it and hold individuals and institutions 
accountable.” Our starting question is: how, and 
under which conditions, do truth claims that are 
embedded in algorithmic systems and associ- 
ated data practices function as a justification for 
a myriad of harms? 
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A way for us to answer this question lies in un- 
derstanding and accounting for how mate- 
rial artefacts—that is to say the instruments, 
devices, and sociotechnical systems—con- 
tribute to an understanding of algorithmic 
systems as objective or scientific in nature. 
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1. Smith, A. (2019, Of these artefacts, bench- 
fe December) Ab mark datasets play a crucial 


Discriminatory Data 


In, Discrimination Out. role in the constitution of 
SHRM. https://www.shrm.org/ the machine learning life 


resourcesandtools/legal-and- 


compliance/employment-law/ cycle. These datasets 

SSeS EE NOLEN ee are used in the training 

criminatory-data.aspx; Richard- 

son, R. Schultz, J., Crawford, and development of ar- 
ae tificial intelligence. They 

predictions: How civil rights 7 ie 

violations impact police data, establish a gold stand- 
predictive policing systems, a” H 

and justice. New York University ard for specific Al tasks, 
Law Review Online 94:192. defining the ideal out- 
Me ane puts of an Al system for 
2. Waseem, Z., Lulz, S., Bingel, aset range of exemplar 
J. & Augenstein, I. (2021) . 

Disembodied Machine Learning: data inputs. Benchmark 

On the Illusion of Objectivity datasets can be under- 


in NLP. https://arxiv.org/ 


Rect itiogd stood as measurements 


for assessing and com- 
3. Benjamin, R. (2019) . 4 _ 
Race After Technology: 7 paring different Al algo 
Abolitionist Tools for rithms. Within Al research 


the New Jim Code. "a0 
Medford, MA: Polity communities, performance 
Press, p.122. on benchmark datasets is 





often understood as indic- 
ative of research progress on a 
particular Al task. Benchmarks are 
the equivalent of IQ tests for algorithms. Just 
as IQ tests are controversial because it is un- 
clear what exactly they measure about human 
intelligence, what benchmark datasets are 
supposed to be measuring about algorithms 
has never been fully articulated. And while the 
role of IQ tests in historically providing justifi- 
cation for white supremacist beliefs is well rec- 
ognised, despite these data infrastructures 
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having significant social impact through the 
dissemination of unjust biases, they remain 
strikingly under-theorised and barely under- 
stood, or even acknowledged, in the public 
sphere. In what follows, we will address bench- 
marks via two definitions which entail, as we will 
see, different types of problems and critiques. 
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«@@ Defined from a purely technical per- 
spective, a benchmark is “a prob- 
_ ae » lem that has been designed to 
Ekembe Ngondi, G. evaluate the performance of 
ae a system [which] is subjected 

ictionary of Computer 
Science, 7th edition. to a known workload and the 
aaa tde performance of the sys- 
niversity Press. = . : 
tem against this workload 
) ismeasured.” The objec- 
tive is to compare “the 
measured performance” 
with “that of other sys- 
tems that have been 
subject to the same 
benchmark test.” 













In order to illustrate 
the limits of a purely 
technical understand- 

ing of benchmark data- 
sets, let’s briefly discuss 
ImageNet, a large visual data- 
set which is used in visual object 
recognition research. Its stated 
objective is to map “the entire world 


THE CASE FOR INTER: 
































of objects.”° The dataset contains more 
than 14 million hand-annotated images, 
and was one of the largest created at the 
time, making it one of the most important 
benchmark datasets in the field of com- 
puter vision. However, as research led by 
Kate Crawford and Trevor Paglen has il- 
lustrated, ImageNet does more than anno- 
tate objects with relatively straightforward 
descriptions (such as “apple”, “round” or 
“green”).° The dataset contains a signifi- 
cant number of categorisations which can 
only be described as depreciative and de- 
rogatory. For instance, a photograph of a 
woman in a bikini is labelled “slattern, slut, 
slovenly woman, trollop” and a man drink- 
ing a beer as “alcoholic, alky, dipsomaniac, 
boozer, lush, soaker, souse.” So, how is 
\ it that morally degrading, misleading, 
and defamatory descriptions shape 
the purportedly objective structure of 
_ the benchmark dataset? 


eh, One explanation lies in the 
5. Gershgorn, D. 
(2017, 26 July) fact that benchmark data- 


The data that transformed . . 
Al ceseanch = ad oo sets are typically perceived 


the world. Quartz. https:// by the machine learn- 
qz.com/1034972/the-data- H ; 

that-changed-the-direction-of-ai- ang community 2 
research-and-possibly-the-world/ purely technical 
devices which 


6. https://excavating.ai/ 





sUsX@lsINI(@OIENS 








6 
6 
T 
= 
ie) 
— 


E 9240 


82 THE 





provide factually objective data. This is 
the case of ImageNet, in which a particular 
label attached to an image is often inter- 
preted as a truth claim about the nature of 
the object or phenomenon depicted. As a 
result, benchmark datasets operate on—and 
reinforce—the assumption that they represent 
some “fact” in the world. A purely technical 
definition of benchmarks also does not take @ 
into account how social, ethical and politi- 
cal factors shape the dataset. Clearly, we 
should question the assumption that tech- 
nical objectivity is somehow embedded in 
benchmark datasets. A different framing 
of benchmark datasets—which takes into 
account the context of the production 
process that shaped them into exist- 
ence—is needed. 




























In this reformulated definition, 
benchmarks can be understood 
as socio-technical measurements, | 
governed by specific norms that, 
in turn, act as standards of 
evaluation. As we can see, 
for example with the 7. Russakovsky, O., Deng, J., 
Su, H.,et al. (2015) ImageNet Large 
ImageNet Large Scale Scale Visual Recognition Challenge. 
https://arxiv.org/abs/1409.0575 

Visual Recognition 

7 - 
Challenge, state 8. Metz, C. (2021, 16 March) 
of-the-art per- The secret auction that set off the race 
formance on the for AI supremacy. Wired. https://www. 

wired.com/story/secret-auction-race-ai- 


benchmark chal- supremacy-google-microsoft-baidu/ 
lenges came to 


be understood as not only a measure of suc- 
cess on the specific formulation of object rec- 
ognition represented in the benchmark, but asa 
»™ much broader indicator of Al research progress. 
Researchers who have produced state-of- 
the-art performance on benchmark datasets 
have gone on to receive prestigious positions 
e _ in large industry labs, or received massive 
\, funding for Al startups.* Such a perva- 
sive view of benchmark datasets as val- 
ue-neutral markers of progress is both 
misguided and dangerous. Benchmark 
datasets can be more appropriately 
understood as tools that institute 
and enforce specific norms, which 
establish normative modes of func- 
tional accuracy, error rates and such. 
In other words, benchmark datasets 
are measurement devices that function 
as a regulative mechanism for assessing 
and enforcing a particular standard in 
a technical setting. When under- 
stood this way, it becomes clear 
that the histories of their pro- 
duction and epistemological 
limits should be thoroughly doc- 
umented. Current benchmarking 
practices reveal troubling moral 
problems, in particular when gold 
» ae standards become petri- 
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and reproduced. In such 
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circumstances, benchmarks normalize and 
perpetuate arbitrary assumptions which 
function as normalization mechanisms. 
















For us, as researchers in the field of A\, to 
address the harmful effects of benchmarks 
understood as normalizing mechanisms, we 
propose cultivating a careful and responsi- 
ble critique which analyses the formation of 
meaning inherent in these technologies. By 
understanding the socio-technical nature 
of dataset production and their contingent 
genesis, we create the conditions 

to stem the myriad harms 
mentioned in our introduction. 

In a word, analysing data- 
sets along technical, social, 
and ethical axes reveals 

their contestable modes of 
construction. Datasets have a 
socio-ethical reality that is distinct 
from their socio-technical dimension. 
As such, it’s possible to recognise 
datasets as the contextual product, 
or contingent outcome, of normative 
constraints and conflicts between 
various stakeholders with visible or 
concealed agendas. Thus, the representa- 
tional harms that we have referenced in 
the context of ImageNet are not simply the 
unexpected and unfortunate effect of pur- 
portedly objective algorithmic systems, but, 
most importantly, they can be traced back 
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to their interpretive origins, that is, the under- 
lying conditions, presuppositions, practices and 
values embedded in dataset creation. 


By placing (distorted and degrading) labels on 
how various beings and objects exist in the 
world, benchmark datasets exert a computa- 
tional power of naming. This power of naming— 
which mimics and recalls, for example, Linnaeus’ 
Promethean efforts in Species Plantarum—oper- 
ates as a power to classify and catalogue all the 
existing objects in the world. By labelling an 
object available to the computational gaze, 
the dataset grants and with- 

draws recognition based on 

the socio-technical assessment of 

the object’s identity. Left unchecked, this mode 
of perceiving and labelling the world amounts to 
accepting and scaling the reproduction and nor- 

malization of representational harms. 


To address this problem, we seek to vali- 
date benchmark datasets by analysing 
their internal modes of construction. 

We aim to do so by using techniques of 
interpretation which establish rules for 

understanding the relation between data col- 


7 — _ lection practices and the 
i, ways in which they shape 
©) models of algorithmic 





.. development. 
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In particular, a key necessary step is perform- , 
ing an interpretive socio-ethical analysis of 
how and when training sets should be used 
in the creation of benchmark datasets. This 
should cover: the disclosures associated with 
the work and research on specific datasets; 
the stakeholders involved and their reflective 
or unreflective intent, data practices, norms 
and routines that structure the data collection, 
and the implicit or explicit values operative in 
their production; the veiled or specific assump- 
tions of their authors and curators; the rela- 
tion between the intended goals of the original 
authors and the actual 
outcomes; and the 
adoption of datasets 
and the related prac- 
tices of contestation by 
subsequent researchers. 
This way, we will be ina 
position to adequately 
identify and interro- 
gate the historical 
conditions of data- 
set production, doc- 
ument their related 
norms, practices and 
axiological hierarchies, 
and thereby both reveal 
and prevent the excesses 
that currently operate in 
the machine learning pipeline. 
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Black feminism is rooted in the knowledge 
that every system and technology has race, 
gender, and class implications. Black 
feminism grows from Black women’s 

experience of living at the intersec- 
tions of race, gender, class, and 
many other axes 
of oppression. 





Black feminists know 
That we 
(OF Talakelmaveam aceon 
Oppression. 
From work 
To the kitchen 
Everything 
Is political. 










No technology is 
truly race neutral. 
Breakthrough research, 
such as that conducted 
by Dr Safiya Umoja Noble 
about the racism and sex- 
ism embedded in search 
engines, has proven 
it. In fact, the very 
claim that algorithmic 
design is race neutral “ 
warrants caution in 
that it is meant to 
reinforce white, West- 
ern, and male norms 
and power. 
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Black death is 

No glitch. 

Black erasure is 

No glitch. 

Black surveillance is 
No glitch. 

Black false positives is 
No glitch. 


New questions must guide the 
purpose, design, development, 
and implementation of A\. 

. Not only are new approaches 
needed to ensure that all 

groups benefit from AI, they 

are also needed to curb “i disillu- 
sionment. Every report of another 


instance of racial discrimination in 
i further fuels rightful Black scep- 
ticism. It is not enough for the goals 
of Al to be non-discriminatory. Al 
must seek to benefit and liberate 
Black women and all historically 
marginalised groups. 





I dream of AI being 
Crafted by Black hands 
And Black dreams. 

If I can sit around the table 

And gush about AI 

With my Mom, Sister 
And Aunties, then 
Pll believe in it. 


Black feminism provides 

a roadmap for improving 

\l, because Black feminist 

politics “actively [com- 

mit] to struggling against 

racial, sexual, heterosex- 

ual and class oppression, 

and sees as [its] particular 

/ task the development of 

integrated analysis and 

practice based upon the fact that the 

major systems of oppression are inter- 

locking.”' Current research on the im- 

pacts of /) ignores those at the inter- 

sections of marginalised identities and 

the differences between historically 

marginalised groups.’ This poetic guide 

can be used to measure the usefulness 

and validity of various Al applications 

with a Black feminist lens. If an Al sys- 

tem cannot pass the muster of these 

questions, | recommend seriously con- 

sidering that it may be AI hype, snake 
oil, or pseudoscience. 
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1. DOES THIS ALGORITHM 
INCREASE INDIVIDUAL, INTERPERSONAL, 
AND COMMUNAL WELL-BEING 
FOR BLACK WOMEN? 





An Kl system is not useful for me 
as a Black woman if it reinforces 
racist and sexist norms. If an,Al 


g system perpetuates the hyper- 
= lisaton of Black 
S soxtla saton'o : ack women, 1. The Combahee River 
q then it does not increase my Collective (1978) A Black 
= well-beina. If an automated de- Feminist Statement. Capitalist 
oe g- 7 Patriarchy and the Case 
cision system defaults to white VOMNUG IN ® CAA 
content, it does not increase @iNENNaaeEeENN 
Pa : ork: monthly review press. 
my individual, interpersonal or 
Si 7 2. Hanna, A., Denton, E., 
communal well- being. If the Smart, A., & Smith-Loud, J. 
algorithm prioritises content (2020) Towards a critical race 
that aligns with colourist, tex- aaaaReeea 
. . . airness. Proceedings of 
turist, fatphobic, queerphobic the 2020 Conference on 
or trans hobic norms then it Fairness, Accountability, 
P . eas oun ITEM AKCIINY Yen arene BLO) 
« does not increase individual, (Bisirupenussmereyo 
a) J 
4 interpersonal or communal 
Ne . 
g well-being for Black women. 


2. IS THIS ALGORITHM 
LESS BIASED THAN 
A BLACK FEMINIST? 


If the AJ system is only better than 
humans because it is slightly less 
racist or sexist, then itis not useful 
for Black women. 


BLACK FEMINISMS? 
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3. SHOULD BLACK FEMINISTS BE HYPED? 


If Black feminists are hyped, then I know indi- 
vidual, interpersonal, and communal well-be- 
ing is going to increase. If Black feminists are 
hyped, then I know it will be beneficial for 
historically oppressed people, because Black 
feminisms are rooted in the knowledge that 
oppressions are tied together. Since oppres- 
sions are tied together, so is liberation. 


4. WHY IS THIS Ai) ASOLUTION 
FOR BLACK WOMEN? 


It is typical in the West to look for technical 
solutions to social problems. But racism, sex- 
ism, and class difference cannot be fixed with 
\! alone. How does this algorithm help Black 
feminists oppose racial, gender, and class op- 
pression? Who is designing this system? Who is 
classifying groups?’ Is this I about bias minimi- 
sation or profit maximisation?’ Is this a job for 
At or ajob that should be given to Black women 
who most likely already know a solution? 


This poetic guide relies on historical 

evidence that: 

If Black women aren’t 
mentioned, that means 

Race Aer ahi, ee tools } Black women will be 

for the new Jim Code. Cambridge, UK: harmed. If Black peo- 

Mie ple aren't mentioned, 


5. See note 2, at p.9. 


CREE YOR DANIDIBYNEG).200 W= Ni 


3. Ibid., at p.10. 


that means Black people will be harmed. If gen- 
der minorities aren’t mentioned, that means gen- 
der minorities will be harmed. If any historically 
marginalised group is not mentioned, that group 
will be harmed. 


LDF 0.4848 


Any marginalised group can ask these (and more) 
questions and it’s important that we do. Current 
analyses of fairness in ff often flatten the nuance 
between—and variety in—Black women’s expe- 
riences.° While flattening identities may make it 
easier for M, it harms Black women and leads to 
further A disillusionment. If the field of A is to 
g2in public trust, it will have to prove its benefits 
to all people. 
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Do we need AI or 

Do we need Black feminisms? 
Liberation should lead, 
Technology should support. 
There is no mechanical solution 
To sin. 
There is only 
The purposeful striving towards 

Justice 
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ties. She is the Policy Research Analyst at Data 
& Society. 
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News of the world’s first toothbrush with 
artificial intelligence arrived in my email 
in January 2017. It was part of the tide of 
announcements accompanying the start of 
the world’s biggest tech trade show, CES; 
an annual inundation that floods journalists’ 
inboxes with the regularity of the Nile, leav- 
ing behind a rich and fertile substrate that’s 
perfect for raising some blog posts. 















It was not the stupidest 
pitch | received that 
year, nor was it the 
first time someone had 
tried to sell me a gadget 
that claimed to have 
“embedded AI” when it 
very obviously didn’t. But 
it sticks in my memory all the same. 
There was something so nonsensical, 
so inadvertently Dada about the phrase, that 
| remember getting up from my desk and mak- 
ing a cup of coffee to process it in mild won- 
der. The world’s first toothbrush with artificial 
intelligence, | reflected, waiting for the kettle 
to boil. The world’s first toothbrush with artifi- 
cial intelligence, | pondered, returning the milk 
to the fridge. 







Now, | freely confess to occasionally writing 
tech stories with ridiculous headlines just for 
the pleasure of lobbing -- scid headline 
into the world like a can- ’ taloupe into 





the Grand Canyon. Any tech journalist who 
says they don’t do this is either lying or has 
forgotten how to have fun. People love to 
dunk on stupid gadgets and tech writers 
love to help them. (Remember: it’s not click- 
bait if you can get the gist of the story from 
the headline alone. If you read, groan, then 
click to find out more about this or that stupid 
gadget out of a sense of morbid curiosity, 
then that’s just a good headline.) 























To write these headlines, most of the time 
you don’t need to do more than state what 
_ itis the gadget does. The whole point of 

~ this sort of coverage—indeed, one sus- 
pects, the whole _ point of this sort of 
gadget—is the self-evident silli- 
ness. The products become more 
enticing the more superfluous 







they are, like a variant of a Veblen good 
in which demand for an item increases 
as it becomes more expensive. | From 
a journalist’s point of view, this usually 
means you can avoid adding snark to the 


headline and let the thing’s idiocy speak for itself. 
People on social media will quote-tweet you 

with delight, pointing out the daftness of 

the product and perhaps adding some com- 
ment about “late capitalism” for good measure. 
All this as if the Victorians didn’t also fill their 
lives with air-conditioned top hats and mechan- 
ical hairbrushes. “Kohler’s smart toilet prom- 
ises a ‘fully-immersive experience” is one such 
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headline | remember with fondness from CES 
2019—the silliness of the semi-submerged pun 
matching, | hoped, the product’s stupidity. 


LDF 0.0766 


It’s still important, of course, for tech jour- 
nalists to write responsibly about even 
idiotic gadgets, and two things should 
be kept * inmind when creating this sort of 
coverage. a First, whether the product in 
question ‘ is so obviously a scam or so 


potentially os’ “ ) 
harmfulthat “(> =, 


covering it in 

any way is unethical. And second, the 
degree of credulity with which you write 
about claims being made about the 
technology. Is this a silly gadget or is 
it an actual con? 


LDR 0.036 












For the Al toothbrush this left me 

with a bit of a problem. (I’d decided 
while drinking my coffee that it was too 
foolish not to write up.) It seemed like a 
real product, no worries on that score, but 
| was uneasy about repeating clcims about 
the device’s “embedded AI”. Would people 
get that this was a stupid thing to say, 
or would they think, given the level 
of ambient hype surrounding arti- 
ficial intelligence, that this was a 
legitimate 
breakthrough of 
some sort? 
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I’d found this dilemma to be a common prob- 
lem writing about Al. The term “Al” itself is so 
rotten with ambiguity, so overburdened with 
varying connotations and expectations, that 
it cannot be trusted to support much linguistic 
weight. Depend on your readers to parse the 
phrase “artificial intelligence” this way or that 
and its meaning will collapse underfoot. This is 
both symptom and cause of the febrile atmos- 
phere that surrounds discussion of Al—a misun- 
derstanding that is productive for generating 
hype and selling snake oil, but not much else. If 
an artificial intelligence researcher, tech journal- 
ist, and lay reader all have different expectations 
when reading those two letters Al, clear com- 
munication about the subject , is always going 
to be difficult. 
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So what to do about 
the toothbrush? As I| read 
through the press release, 
Googled the company that 
made the gadget and scoured 
their website for something 
approaching detailed tech specs, 
it seemed—as expected—that 
what was being passed off 

as “artificial intelligence” was 
just a teaspoon of embedded 
computation: the thing was 
only smart compared 
to rocks. 
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What it did was use sensors 


2 to record when it was used, how 

A long for, and, roughly, the parts of 
the mouth covered by each brush- 
ing. It refined this data into graphs and 
charts and sent the result- ing insights 
to a smartphone app. It then delivered 
revelations along the lines of “brush 

S more” or “brush less” depending on 

= howmuch you brushed. It was the sort 

A of information that a stopwatch could 
provide without mak- ing any claims 
to embedded intel- ligence. A little 
further Googling even revealed that 
the gadget’s cre- ators had announced 
a near-identical product in 2014. The 
only major (= change was that they’d 
dropped the . term “smart” in favour 
of the more fashionable “/\”. It seemed 
this was just another company climbing 

~ aboard the artificial intelligence hype train, 

but what else had I expected? 

ie) 

~ Inthe end, | wrote a short little blog post, 


barely 300 words, with the head- 

line “This smart toothbrush claims to 

| haveits very own ‘embedded Al’”. | hoped 
this formulation—the scare quotes and 
“clasms”—would neutralize any poten- 
tial deception, but I’m not sure it really 
did. Later in the day, a little annoyed I’d cov- 
ered the thing at all, | wrote a longer arti- 
cle castigating superfluous “il gadgets” in 
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general. More searching dug up “.jl-enhanced” 
alarm clocks, TVs, beds, headphones, washing 
machines, dishwashers, fridges, and more— 
each product trading off the same ambiguity 
surrounding artificial intelligence to sell basic 
digital functionality as something exciting and 
new. Plenty of these gadgets attracted press 
coverage—some critical, some credulous—but 
browsing through it all | felt that no matter what 
had been written, it only added to the confusion. 
Taken together it all seemed like epistemolog- 
ical chaff—meaning thrown to the winds. The 
world’s first toothbrush with artificial intelligence 
had only ever been a single point in a scattered 
and chaotic melée. 
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James Vincent is a senior reporter for The Verge 
who covers artificial intelligence and other things. 
He lives in London and loves to blog. 
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Whether it’s forming a scientific hypothesis or 
seeing deities burnt into toast, humans are hard- 
wired to find patterns in the world. We are adept 
at making sense of the chaotic or unknown from 
information we already know. 


LDF 0.0316 


This can elevate understanding 

'’ asmuchasitcan mislead. One con- 
sequence is our tendency to pro- 
ject the human onto the artificial, a 

| phenomenon called anthropomor- 
‘phism. Similarly, ascribing animal-like 
_, qualities to non-animals is zoomor- 
_ phism. This can result in curious 
& behaviour. People give names to 
i their robot vacuums, which 
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’ they hate to see struggling in 
) corners. Apple users thank Siri 
when she answers a question. 

Acar owner feels subject toa 
personal attack by their bro- 
ken-down car. 


LG 5.5246 


Advanced technologies 
e=,, are particularly ripe 
' for this sort 
of misperception, as operating mechanisms are 
often opaque, capabilities are increasingly human 
or animal-like, and the results can seem magical 
to those who do not have insight into the design. 
Indeed, as soon as room-sized computers were 
first developed, they were commonly referred 
to as “electronic brains”. Obviously, they were 
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nothing like brains, but the phrase 
sounded super futuristic, gave the public 
a grasp of their functions, and grabbed 
more attention than something bland, 
like “Electronic Numerical Integrator 
and Computer”. 


While metaphorical descriptions which equate 
machine capabilities with human traits may ini- 
tially appear harmless, they can lend more agen- 
cy to the technology than is warranted. If you 
read the headline “Scientists Invent Giant Brain” 
you would imagine much about that device based 
on your understanding of a brain and its func- 
tions. When we ascribe human traits to technol- 
ogy, those traits often come with assumptions of 
abilities which may not be true. For example, even 
if an Al system produces human-like text, it can- 
not be assumed that it also has other capabilities 
associated with language, such as comprehen- 
sion, perception or thought. Nevertheless, most 
of us do this unconsciously and automatically. If 
we are aware of our tendency to anthropomor- 
phise, it’s easier to notice when this sort of bias 
kicks in, making us recognise advanced tech- 
nology for what it is. 


Equally, concepts drawn from sci-fi are often 
used as a shorthand to describe (and sometimes 
overrepresent) the capabilities of technology. 
Probably journalists do this because it creates a 
quick and familiar image in the reader’s 

mind. (More likely, it is to chase clicks.) 
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_ One example is when self-balancing skate- 
» boards were called “hoverboards” in refer- 
ence to the movie, Back to the Future. Since the 
device worked using wheels which were firmly 
planted on the ground, no hovering was involved 
at all. Still, “hoverboard” was a better buzzword 
than, say, “self-balancing transport”. The impli- 
cation was that sci-fi technology was here now, 
even though it was clearly not. 








Something similar has happened with artificial 
intelligence. There are two important categories 
of Al. Artificial intelligence (Al), which is a type of 
machine learning that carries out a specific task, 
and artificial general intelligence (AGI), which is 
a system that is capable of doing any non-phys- 
ical task as well as, or better than, a human. /\l 
in a car can determine whether or not there is a 
stop sign in an image captured by a camera. An 
AGI can do the same while also being the CEO 
of the company that builds the cars and writing 
flawless poetry. 


Unfortunately, in media reporting, Al is some- 
times treated like AGI. Take the example of 
an \\ chatbot developed by Facebook which 

sometimes formed incomprehensible English 
output when two such Als “spoke” to each other. 
This was an artefact of the system attempting 
to find a more efficient way to communicate, a 
sort of shorthand. Here’s an example of the two 
chatbots negotiating how to split up basketballs 
between them: 
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 ) * Bob: 1 can 11 everything else 


PANU Cepan of-1 UI at-htcomyAs) KOM KOO OloM Kom enlon (Om pelemcemnslemne) 
100Lom KOM Alem KO ealom Kem aelomne) 


Bob: you i have everything else 


Alice: balls have a ball to me to me to me to me 
to me to me to me to me 


This kind of semantic soup became such an 
obstacle to the intended function of the sys- 
tem, they stopped research. 


A human analogue to this Al chat might be 
drawn with military acronyms or scientific 
terms, which make sense to those within a 
particular communication system, but not to 
those outside it. However, alarmist headlines 
evoking images of a movie hero “killing” an Al 
poised to destroy humanity were used to report 
the incident: “Facebook engineers panic, pull 
plug on Al after bots develop their own language”. 


















This kind of sensationalism leads to misun- 
derstanding and fear, when the reality of Al is 
often far more mundane. 
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Another phenomenon which contributes to the 
misrepresentation of .l’s capabilities is “cherry 
picking”, whereby only the best system out- 
puts are shared. The rest are hidden or omit- 
ted, and no mention is made of the winnowing 
process. Sometimes this is intentional, such as 
when artists generate Al outputs for artworks. 
Other times this can be a tactic used to inten- 
tionally overrepresent capabilities. This might 
be done to make systems appear better than 
they are, to gather more attention and support, 
or to make the system seem more interesting 
to potential investors. 


Within an artistic context, where an artist 
works with the system to make a creative 
output, “cherry picking” is more akin to 
curation. For example, in one of my own 
artworks, | trained an il on fortune cookie for- 
tunes so it could generate new ones. | then 
chose outputs | liked and had them printed 
and enfolded in actual fortune cookies. 








“Your dreams are worth your best pants 
when you wish you’d given love a chance” 


“T am a bad situation” 
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“Your pain is the essence of everything and the 
value of nothing” 






“Today, your mouth shut” 


F2irly unusual fortunes which 

make sense, if you don’t think 

about them too hard. Many others 

which | left out didn’t catch my eye ini- 

tially. Yet they kind of have meaning, if you 
bend your mind a bit: 


“There is a time when you least expect it” 
“Learn to take on the ceiling” 


“Emulate what you need to relax today” 


“You are what you want” 


Then there are the 
really nonsensical ones: 


“Nappies and politicians need to listen to” 


“You are a practical person with whom you 
exchanged suggestive glances?” 


e 
ING) :; i 


“You will be out of an ox” 


“The greatest war sometimes isn’t on 
the ground even though friends flatter you” 


Cherry pick- 

ing without dis- 

closing that you 

are doing so leads 

to a misrepresen- 

tation of current /\ 

systems. They are 

already impressive and 

improving all the time, 
but they are not quite 
what we see in sci-fi. 
So next time you encounter or read . 
about an &, ask yourself the following ques- 
tions: Am I projecting human capabilities onto 
it which may be false? Are sci-fi ideas and pop- 
ular media clouding my perceptions? Are the 
outputs cherry picked? We need to keep in 
mind that if current Ai is to AGI as “self-bal- 
ancing transports” are to hoverboards, we are 
still a long way from getting off the ground. 
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Alexander Reben is an MIT-trained 
artist and technologist who explores the 
inherently human nature of the artificial. 
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116 USES (AND ABUSES) OF HY‘? 


Hype has 
always been 
a part of my 
career. As a sci- 
ence and technology 
researcherlspend 
time following the 4 
science startup 4 
receivingend ¢ 














writerand ° 

alot of my 
technology and 
scene. I’ve been on the 
of bombastic pitches, 





I’ve helped people desperate to get 
noticed, and | wrote a book on how hype 
obscures the future. | believe that 
until we reckon with the dual nature of 


* hype, snake oil and pseudosci- 
/ ence will persist in A\, and science 
and technology more generally. 


Reactions to hype range from excite- 
ment at the genuine progress that /\| has made, 
to scorn at lame sales pitches. The word has a 
variety of definitions too, depending on who you 
ask. Some see it as unfaithful to the truth, a lie 
of sorts. Others consider it as merely an ultra- 
subjective form of expression, a form of “fair play” 
marketing, as it were. 


To me, hype is a tool for capturing attention. 
Hype is exaggerated publicity which inflates 

expectations and prompts emotions such as 

excitement or fear, which can stop people in 
their tracks. It’s a little like a magic show. By tug- 
ging on the emotions, hype causes people to lend 
their ears, eyes, and brain waves to ideas, claims, 













































and sales pitches in an uncritical manner. The 
tricks are wondrous, but they’re entertainment. 


The use of hype comes with a warning label 
though. Forget that it’s entertainment and 
it becomes lying, usually for nefarious ends. 
Furthermore, hype can result in accidental fool- 
ing. Out of context, hyped messages can very 
easily be misinterpreted. The originator may 
not have set out to trick or deceive, however, 
the fact remains that because of its spread, 
the language used, or the cultural context, 
hype can be misinterpreted as truth. 


Yet the role of hype is more significant than 
simply capturing attention at all costs. It 
eases the complicated, arduous, and expen- 
sive journeys between research lab, office 
meeting room or simply someone’s brain, 
out to the market and the public. It lubri- 
cates, bridges, and facilitates. Hype can be 
a strategic tool for selling ideas, products or 
political campaigns. It helps attract people, 
pennies, policies, and partnerships. 


Hype is, in and of itself, a creator of value— 
regardless of the substance behind it. 


Hype attracts attention, money, good employ- 
ees, and social capital. Startups want the 
attention of investors. NGOs want to 
attract funding. Thought leaders want 
to lead thought. Researchers want 
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better- 
funded labs. 
Universities want to 
be higher on the league tables. Companies 
~ want more clients. To align yourself with 
“innovative” narratives gives you a head 
over the competition. 








That’s why hype is unlikely to disappear. It is 
genuinely useful for generating attention—a 
scarce resource in today’s attention econ- 
omy. But where there’s opportunity, there’s 
opportunists, and so the existence of /\i snake 
oil is unsurprising. Its prevalence, though, is 
dependent on the rest of us not reckoning 
with the problematic, dual nature of hype. 
This needs to happen on two levels. 


First, those on the receiving end of hype need 
to collectively recognise the power, mechan- 
ics, and sheer volume of it. The sole purpose 
of a hyped-up message is to grab attention. 
So when those on the receiving end do not 
scrutinise the message further, a lack of con- 
text, or the absence of deeper understanding 
about the topic can easily result in accidental 
fooling. There are also places and circum- 
stances where hype must be checked at 
the door entirely—otherwise the public, 
journalists, investors, and government 
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ministers will continue to find separating snake 
oil from legitimate claims extremely difficult. 
Hype about technology, for instance, has no 


place in serious policy deliberations about how 
to regulate technology. 


Enhanced crit- 
ical thinking and 
a better under- 
standing of how 
hype and mar-¢) 7 L 
keting “work” ; 4 


can help reduce 
hype’s problem- 
atic impact. Its power, 
after all, is in its illu- 
sion, so seeing it for what 
it is can help stop it in 
its tracks. 


Still, that’s not enough to stem 
or limit the volume of hype that is 
produced, especially in Al. 


Secondly, those of us who work in sci- 
ence and technology—as builders, poli- 
cy-makers, commentators, researchers, 
investors—must reckon with our own 
complicity in the use of hype. This 
requires introspection, acceptance, 





= 






and possibly a pinch of low-level shame. Hype 
helps projects move forward. We might not use 
those words, but we know it to be true. 


LDF 0.0428 


We could argue that hype is just a tool used on 
the road to greatness, that this lubricant is just 


S a reality of getting 
& through that complicated, arduous, and expen- 
sive idea-to-market journey. We can justify its 
wielding like advocates of placebos justify 
lying to unsuspecting patients. The cost of 
accidental fooling that hype comes with is 
worthwhile as it’s the only way to get stuff 
done in the dog-eat-dog world of the 
innovation industry. The ends justify the 
means when saving the world is at stake. 




















This pragmatic view neglects the dual 

nature of hype. The very lubricant that 
eases ideas through complex systems is 
what allows snake oil to make it to mar- 
ket. Hype is useful to capture attention 
in today’s attention economy, but snake 
oil is an inevitable by-product. Those who 
generate, amplify or simply cash in on 
the hype surrounding “! inevitably also 
create the conditions in which snake oil 
vendors and pseudoscience can thrive. 
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Lack of 
care characterises a 
lot of the problems around hype in science and 
technology. Those on the receiving end who do 
not listen might be ignorant, flippant or time- 
poor. Then there’s the fact that those creating 
the hype are often willingly complicit; carelessly 
wielding a tool which makes them look credible. 
If we reckon with these two forces, snake oil will 
start to have less success. 


Beyond that, what’s needed is a genuine culture 
change in technology and science. The pres- 
sure to publish at all costs in order to progress 
in academia, the precarious working conditions 
in journalism, and the growing competition for 
funding are all systemic forces that create con- 
ditions in which problematic hype can thrive. This 
culture often equates value with attention, and 
links status and fame to trendy topics or cults of 
personality, rather than proof of positive social 
impact. Publish or perish should itself die. 


Perhaps what we need is more boring 4. More 
cautious product pitches. More thought 
leaders who communicate uncertzinty 

about their own thoughts. 





Snake oil in,Al will thrive until the vari- 
ous forms of hype that keep it trendy are 
deeply and unashamedly reckoned with 
by those who arguably stand to benefit 
most from no-one saying anything at all. 


Gemma Milne is a Scottish science and 
technology writer and PhD researcher in 
Science & Technology Studies at University 
College London. Her debut book is Smoke 
& Mirrors: How Hype Obscures the Future 
and How to See Past It (2020). 
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13th century. 
it was sciid, the scien- 
. tist and philosopher Albertus 
Magnus built a talking head. 
This contraption, skilfully 
manufactured with hidden 

| | wheels and other machinery, 
spoke so clearly that it fright- 
ened his pupil, who smashed it. ! 


1. Anon (1627) | 
The famous 


iene Like us, the people of the Middle 
* London: A.E. for 
P= Francis Grove. Ages told themselves myths 


fe. about technology. One 


| 
126. 





particular mythic strand 
involved talking heads. ‘~ 
Like today’s systems, these “ » 
heads received inputs from “\\ 
their builders and offered out- / 
puts that told of things to 
come. However, the heads 
failed as predictive systems. “)\ 
The tales warned of automat- “\_ 
ed data operations gone wrong. 










Such fables probably resonated 
with their audiences for several 
reasons. One is that although 
there were many known exam- 
ples of automata that were 
capable of moving or perform- 
ing other functions, the pos- ~“ 
session of avoice—the signofa_ ~ 
reasoning soul—crossed a significant Ss » < 
threshold. Another is that these sys- “" 
tems occupied an ambiguous, or some- q > 


times overtly transgressive, place in the 
taxonomy of technology. 


In the Middle Ages, technology—the “science 
of craft”—embraced three categories, which 
we might for convenience call exoteric, eso- 
teric, and demonic. 


Exoteric technologies, such as mills, bows, 


cranks, and wheelbarrows, operated in ac- 
cordance with physical laws which were well 
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understood and clearly 
visible to all. They were 
uncontroversial. Esoteric 
technologies, on the other hand, occupied the 
space of so-called “natural” magic. This involved 
secret or hidden mechanisms, which were not 
commonly known or understood. Those who 
exploited these mechanisms argued that 
they were nonetheless natural, because they 
operated through certain intrinsic features 
of the universe. 





















The workers of natural magic construed the 
universe in terms of analogies. They perceived 
relationships between heavenly and earthly 
bodies, between stars and comets, animals 
and plants, and they believed in a cause and 
effect straddling these analogies. Not unlike, 
perhaps, the data analysts of today, they 
built their universe through proxies. 


The danger in the practice of natural magic, as 
witnessed in writings and controversies across 
the centuries of the early modern period, was 
that the line dividing it from the third category 
of technology, demonic magic, was highly con- 
tested. Demonic magic comprised any activi- 
ties for which no natural explanation 
could be found, and which there- 
fore had to be the work of demons, 
persuaded by scholars via ritual 
to do their bidding. Martin Delrio, 
for example, recounting the fate 
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of Albert’s head in his Investigations into 
Magic (1599), concluded that the head 
could not be the product of human ingenu- 
ity. This was impossible, he scid, for such 
human-manufactured idols “have a mouth 
but speak not”. ” 


Therefore it was a demon that spoke. 


inthe absence 
of plausible 


explanation, 
2. De’ Corsini, M. (1845) Rosaio della there was a 
vita: trattato morale. Florence: Societa 
Poligrafica Italiana. black box 7 and 


3. Delrio, M. (1657) Disquisitionum in the box was 
magicarum libri sex. Cologne: thoug ht to be 
Sumptibus Petri Henningii. a demon The 
story of Albert’s 
head is about a scientific— 
~ literally, knowledge-making— 

endeavour which ended in the 
construction of an object of horror. 


Before Albert there was Gerbert of Aurillac. 
® Gerbert, it is recounted by the historian 
William of Malmesbury, constructed his 
' head at the apex of a career investigating 
&. computational and predictive systems.’ 
_ He had mastered judicial astrology, the 
analysis of the effects of the movements 
of heavenly bodies on human lives. He 
had “learned what the singing and the 

flight of birds portended”. And as a 
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mathema- 

tician he had 

established 

new ways of 

manipulating 

the abacus, develop- " 
ing for it “rules which are 
scarcely understood even — 
by laborious computers”. 


The head was capable of 
responding in binary to a 
question, answering “yes” 
or “no”. Butits responses, 
which lacked context and | 
strictly adhered to nar- 
rowly defined inputs, were 
misleading. Gerbert, who 
lived in Rome, had asked 
the head whether he 
would die before visit- 

ing Jerusalem. The head 
replied that he would 

not. But it failed to 

specify that there was 

a church in Rome of that 
name, where Gerbert did 
indeed go. Soon after, he died. 


Likewise, it was said that in 
Oxford in the 13th century, 
the philosopher Roger Bacon 
built a bronze talking head 
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as a component in a defence 


system which would ren- __ 

der England invulnerable.* 

Yet after building the head, / 

he fell asleep, leaving aser- | 7. ae a 
vant to monitor it, where- | Chronicle of the Kings 
upon the head uttered mo London: Henry G. 
three obscure phrases 

and exploded. The serv- 
ant, who was blamed for 
the debacle, argued that 
a parrot with the same 
duration of training 
would have spoken 
more sense. Bacon 
himself was convinced 
that the head had 
something profound 

to communicate. 

What this was, no 

one would ever 

know, because 

only the ma- 

chine’s builder , 

was able to - 
interpret it. To 

those charged 

with its man- 

agement, its 

findings 

were 

opaque. 
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A story told a 
of Bacon’s = 
teacher, Robert 
Grosseteste, runs 
parallel. He, too, 
built a head, and 
worked on it for seven 
years. But his concen- 
tration flagged for 30 
seconds as he sought to 
finish the task. A lapse 
in the internal mecha- 
nism or the underlying 
data? We are not told. As 
a result, he “loste all that 


he hadde do”.> 5. Ed. Macaulay, G. C. (1899) 
The Complete Works of 
L John Gower. Oxford: 
These fables, centuries old, Clarendon Press. 


offer us patterns which we 
may find familiar: allegories 
of modern encounters with 
predictive systems. Gerbert 
failed to appreciate how his sys- 
tem’s outputs were conditioned 
by unperceived ambiguities in its 
inputs. A flaw in the underlying dataset 
produced misleading results. Bacon left 
his system running under the management ¥ 
of someone not trained to understand its 
outputs. Consequently, nothing useful was de- 
rived from them. Grosseteste’s machine failed 
owing to a very small programming discrep- 
ancy, a mere half-minute’s worth in seven years. 


132 TALKING HEADS 













The effects of this 

miscalculation, at 

scale, rendered the ma- 

chine useless. Whether or 

not Albert’s black box was 

effective, the tale does not 

tell us, but its inexplicabil- 

ity proved a fatal barrier to its 
acceptance. 


In these myths, the creators of 

these heads dreamt of the future. 

In these fables of fziled predictive 

systems we discern a prediction of 
our present. 


Crofton Black is a writer and investigator. 

He leads the Decision Machines project at 

The Bureau of Investigative Journalism. He 

has a PhD in the history of philosophy from 
the Warburg Institute, London. 
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Until the 20th century, faces could only be 
seen through the human perceptual sys- 
tem. As face recognition and face analysis 
technologies developed throughout the 
last century, anew optical system emerged 
that “sees” faces through digital sensors 
and algorithms. Today, computer 
vision has given rise to new ways of 
looking at each other that build on 
19th century conceptions of pho- 
tography, but implement 21st cen- 
tury perceptual technologies. Of 
course, seeing faces through mul- 
ti-spectral modalities at superhu- 
man resolutions is not the same as 
seeing a face with one’s own eyes. 
Clarifying ambiguities and creat- ' 
ing new lexicons around face rec- | 
ognition are therefore important 
steps in regulating biometric sur- 
veillance technologies. This effort 
begins with exploring answers to 
a seemingly simple question: what 
is a face? 
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The word “face” gener- 
ally refers to the front- 
most region of the upper- 
most part of the human body. 

However, there are no dictionary 
definitions that strictly define where the 
face begins or where it ends. Moreover, to 
“see” hinges on an anthropocentric defini- 
tion of vision which assumes a visible spec- 
trum of reflected light received through the 
retina, visual cortex, and fusiform gyrus that 
is ultimately understood as a face in the mind 
of the observer. Early biometric face analysis 
systems built on this definition by com- 
bining the human perceptual sys- 
tem with statistical probabilities. But 
the “face” in a face recognition sys- 
tem takes on new meanings through 
the faceless computational logic of 
computer vision. 


Computer vision requires strict defi- 
nitions. Face detec- 
tion algorithms de- / 
fine faces with ex- 
actness, although 
each algorithm may 
define these parame- 
ters in different ways. 
For example, in 2001, 
Paul Viola and Michael 5 Viola, P. & Jones, MJ. ue) NEES 

. eal-Time Face Detection. International 
Jones' introduced the Wma of Computer Vision 57, 137-154. DOT: 
first widely-used face 10.1023/B: VISI.0000013087.49260.fb 
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detection algorithm that de- 
fined a frontal face within a 
square region using a 24 x 24 
pixel grayscale definition. The 
next widely used face detec- 
tion algorithm, based on Dalal 
and Triggs’ Histogram of Ori- 
ented Gradients (HoG) algo- 
FUaAE Ae aRRD rithm,’ was later implemented 
Koamrmkecatie@aaa in dlib> and looked for faces 
RnMnrnnTaes at 80 x 80 pixels in greyscale. 
em eTeceROeee Though in both cases images 
DOT: 10.1109/CVPR.2005.177 could be upscaled or down- 
3. https://github.com/davisking/dlib scaled, neither performed 
well at resolutions below 
40 x 40 pixels. Recently, 
convolutional neural net- 
work research has redefined 
the technical meaning 

of face. Algorithms can 
now reliably detect faces 
smaller than 20 pixels in 
Deity cn, Hac. height,’ while new face rec- 
IEEE Conference on Computer og nition datasets, such as 
Faery — TinyFace, cim to develop 
CVPR.2017.166 low-resolution face recog- 

5 hups://qatl ; _ nition algorithms that can 
tinyface.github.io recognise an individual at 
around 20 x 16 pixels.° 


Other face definitions 
include the ISO/IEC 19794-5 
biometric passport guidance 
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| published a recommendation for using between 


for using a minimum of 120 
interocular (pupil to pupil) 
pixels. But in 2017, America’s National 
Institute of Science and Technology (NIST) 











70-300 interocular pixels. However, NIST also 
noted that “the best accuracy is obtained from 
faces appearing in turnstile video clips with 
mean minimum and maximum interocular dis- 
tances of 20 and 55 pixels respectively.”° More 
recently, Clearview, a company that provides 
face recognition technologies 
to law enforcement agencies, 


clciimed to use a 110 x 110 (OM LH OSH /AWAN AWAY Oh 7) 
pixel definition of a face.’ ee Pe ane 

. 7 recognition-vendor-test-frvt- 
These wide-ranging tech- ongoing 


nical definitions are already cary; oe : 

7 . Clearview AI’s founder Hoan 
being challenged by the face Ton-That speaks out [Interview], 

; https://www.youtube.com/ 

masks adopted during the Wratch?v=q- 1DR3POR Aw 
Covid-19 pandemic, which have 
led to significant drops in face 
recognition performance. 


The face is nota single biometric 
but includes many sub-biomet- 
rics, each with varying levels of 
identification possibilities. During 
the 2020 International Face 
Performance Conference, NIST 
proposed expanding the concept 
of a biometric face template to 
include an additional template 
for the periocular region,’® the 








region not covered by 
a medical mask. 
This brings 





into question 
what exactly ‘ s.nttpsi) & 
“face” recogni- www.nist. 


gov/video/international- 


tion is if itis only ana- Wrcsrsriomrnces 
lysing a sub-region, or eines 

sub-biometric, of the part-1 

face. What would peri- 

ocular face recognition 

be called in the profile view, 

if not ocular recognition? The distinction 

between face recognition and iris recogni- 

tion is becoming increasingly thin. 


Ata high enough resolution, everything looks 
basically unique. Conversely, at alow enough 
resolution everything looks basically the 
same. In Defeating Image Obfuscation with 
Deep Learning, Macpherson et al. found 
that face recognition performance dropped 
between 15-20% as the image resolution 
decreased from 224 x 224downto 14 x 
14 pixels, using rank-1 | and rank-5 metrics 
respectively, with a dataset size of only 
530 people. But as the number of iden- 
tities in a matching database 
increases, so do the | 
inaccuracies. Million- 
scale recognition at 
14 x 14 pixels is sim- 
ply not possible. 
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Limiting the resolution 
of biometric systems used for mass 
surveillance could contribute profoundly 
towards preventing their misuse, but 
such regulation requires unfolding the 
technical language further and making 
room for legitimate uses, 
such as consumer appli- 
cations or investiga- 
tive journalism, 
while simultaneously 
blunting the capabilities 
of authoritarian police or 
rogue surveillance companies. 
Calls to ban face recognition would be 
better served by replacing ambiguous ter- 
minology with more technically precise lan- 
guage about the resolution, region of interest, 
spectral bandwidth, and quantity of biom- 
etric samples. If law enforcement agencies 
were restricted to using low-resolution 14 x 
14 pixel face images to search agcinst a data- 
base of millions, the list of potential matches 
would likely include thousands or tens of thou- 
sands of faces, making the software virtually 
useless, given the operator would still have to 
manually sort through thousands of images. In 
effect this would defeat the Pur 
pose of face recognition, a ag 
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whichis tocompress | the search space 
into a human-scale — dataset in order to 
find the needle in the haystack. Severely re- 
stricting the resolution of a face recognition 
system means that searching for a needle 
would only yield more haystacks. 
































In 2020, a project by researchers at Duke 
University, called PULSE, _ showed that 
the restricted per- ) ceptual space 
of a16 x 16 pixel face [Rainy] image allows 
for wildly different |Raigew identities to 
all downscale to per- ceptually in- 
different images.’ The . project faced 
criticism because it - also showed 
that up-sampling a low-resolution 
image of Barack ' Obama produced a 
high-resolution image of a light-skinned face. 
Nevertheless, the work confirmed the technical 
reality that two faces can appear identical 
at 16 x 16 pixels, but resemble completely 
different identities at 1024 x 1024 pixels. 
As image resolution decreases so too does 
the dimensionality of identity. 









Unless a computational definition 
of “face” can be appended to 
the current language around 


42  WHATISAFACKy. 






biometrics, the unchallenged ambi- 
guities between a 16 x 16 pixel face 
and a 1024 x 1024 pixel face will likely 
be decided by the industries or agen- 
cies that stand to benefit most from 
the increasingly invasive acquisition 
of biometric data. Moreover, better 
regulatory definitions of “face” that 
include specific limits on the reso- 
lution of face imagery could help 
limit the potential for face recog- 
nition technologies to be used for 
mass surveillance. The monitoring 
and tracking of our every pub- 
lic move—at meetings, in 
classrooms, at sport- 
ing events, and _— 
even through @ 
car windows— | 
is no longer 
limited to law enforce- 
ment agencies. Many indi- 
viduals are already scraping the 
internet in order to create their 
own face recognition systems. 


A better definition of “face” in 
recognition technology should 
not be limited to sampling res- 
olution, but should also include 
the spectral capacity, duration of 
capture, and the boundaries of 
where a face begins and where 
















it ends. As the combi- 

natory resolution of a 

face decreases, so does 
its potential for mass sur- 
veillance. Limiting resolu- 
tion means limiting power 
and its abuses. 









Adam Harvey is a researcher 

and artist based in Berlin. 
His most recent project, 
Exposing.ai, analyses the 
information supply chains 
of face recognition 
training datasets. 
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When | was a content moderator at Google 
in the mid 2010s, there were days | wished a 
machine could do my job. It was usually on the 
boring days spent scanning through monot- 
onous copyright complaints with thousands 
of URLs, or the bad days clearing out queues 
of child sexual abuse content (they would 
always refill by the next day). 


LG 3.3473 


Content moderation is difficult work, often 
exciting but occasionally damaging to the 

soul. The platforms | moderated were (and still 

are) used by billions of people across the world 

as libraries, news sources, and public squares, 

but they were not built for these purposes. 
Rather, these platforms were designed to 

; y incentivise user engagement, content shar- 
\_ ing, and mass data collection—objectives that 
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i often made moderating harmful content feel like 
throwing cups of water on a raging inferno. When 
| did this work, asmall number of automated tools 
helped prioritise and flag certain types of content 

_ forreview, but human moderators like 

me were ultimately charged with 

putting out the daily blazes. Over 
the years | worked at Google, 
the copyright complaints grew 
longer and the queues of trau- 
matising content continued 
to grow. Surely, | wondered, 
only the increased use of | 
could help moderate these 
platforms at scale? 
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Nearly a decade later, 
the heads of large tech 
firms and global poli- 
cymakers share the 
same vision. In the 
wake of high-pro- 
file incidents like 
the Christchurch 
mosque shooting 
and the Rohingya 
genocide in 
Myanmar, policy- 
makers across the 
globe have called for 
tech firms to remove hate 
speech, terrorist content, 
and other forms of harmful speech at 
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increasingly breakneck speed. In response, tech 
platforms have invested heavily in Al-based mod- 
eration tools to identify, predict, and remove 
harmful content within minutes—if not at © 
the time of upload. Called before the US 
Congress in 2018 over election misinforma- 
tion, Mark Zuckerberg declared that “over the 
long term, building Al tools is going to be the 
scalable way to identify and root out most of 
this harmful content.” Facebook now boasts 
that 94.7% of hate speech removed from 
their platform in the third quarter of 2020 
was proactively identified using their auto- 
mated tools. Has the dream of automated 
moderation come true? Or do statistics like 
the ones above mask deeper problems 
with platform moderation? 
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Sadly, auto-— 
mated moderation 
tools have become yet 
another example of the 
misplaced hope in artifi- 
cial intelligence to solve 
complex human prob- 
lems. By framing online 


TSS W uY AUTOMATED CONTENT 


E 40460 


safety as simply a matter of moving from 
“human to automated”, tech firms and 
policymakers risk exacerbating known 
accountability and transparency issues 
with platform moderation policies while dis- 
tracting themselves from more urgent ques- 
tions around platform design features and 
business incentives. The hype around auto- 
mated moderation tools has largely over- 
looked their technical limitations, the 
hidden labour of human moderators, { ) 
and the increasing opacity of plat- 
form decision-making. 


















The technical limitations of automated 
moderation tools are well known to 
research and civil society communi- 
ties. Notoriously bad at identifying the 
nuance and context of online speech, 
these systems routinely fail to identify 
whether a video constitutes illegal copy- 
right infringement or lawful parody, or 
whether : 

a post 
witha 
racial slur is written by a victim of a hate 
crime or their asscilant. Due to their reli- 
ance on historically labelled content, these tools 
fail to keep pace with the constant evolution of 
human language, suchas the shifting codewords 
used in QAnon misinformation campaigns. Some 
systems exhibit serious issues of language bias— 
for example, researchers have found Google’s 
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Perspective API 
tool, which uses 
machine learning to 

predict the “toxicity” of 
certain content, penalises con- 
tent written in African American 
Vernacular English.' Addressing these 
issues would require not only a para- 
digm shift in Al research butafunda- | Sap, M 
mental reconstitution of the labour (ems 
4 H . - Gabriel, S., 
forces designing these tools to incor- |r 
porate more diverse perspectives. Smith, N.A. 
(2019) The 
. - Risk of Racial 
The torrent of online misin- Bias in Hate 
formation sparked by the __ aemnetee 
. roceedings of the 
Covid-19 pandemic laid 57th Annual Meeting 
these limitations bare like & ER ea eae 
for Computational 
never before. Not trust- Linguistics. DOI: 
ing contracted moderators cies 
to take corporate 

devices home and 

work remotely, 

large plat- 
forms like 



























Facebook and 
YouTube resorted 
to fully automat- 
ing many modera- 
tion decisions. The 
results were alarm- 
ing. Child exploita- 
tion and self-harm 
removals on Facebook 
fell by at least 40%, 
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pages reporting factual Covid-19 infor- 
mation were misidentified as mis- 
information, and YouTube appeals 
of wrongfully removed 
accounts skyrocketed.’ 
Rather than improving @ayEvEN 
outcomes, the results iintmeewn 
highlighted how auto- - rane 
mated tools can worsen humans stopped 
thealready inadequate (Uiitiaen 
status quo of platform [iene 
moderation practices. “Ws oa 
politico.eu/ 
The decision to jettison — 
aceDOoo. 
human moderators dur- content- 
ing Covid-19 also reflects the |Riieaenay 
changing role of the modera- 
tor as platforms have grown in 
size and scale. In my time doing 
this work, most moderators were 
full-time employees respected 
as trusted experts in particular 
regions or products, encouraged 
to deliberate on tricky edge cases. 
Today, tech platforms increasingly 
treat moderators as an expend- 
able and invisible labour source 
whose decisions are used as fuel 
to train automated moderation 
tools. The vast majority of moder- 
ators at large platforms today are 
temporary contract labourers out- 
sourced from third-party agencies, 
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in 





countries 
that include -_: 
India, the Philippines, 
and Malaysia. They are 3. Roberts, S. T. (2019) 
low-paid, relative to full- iQuitiweramenncn 
time employees of large Raat Mea 
tech firms, and lack the CT: Yale University 
same career advancement rcs. 
opportunities and access 
to high-quality mental | 
healthcare that is neces- sary for 
the traumatising aspects of this 
work.’ These moderators are hired, 
constantly assessed, and fired on the 
basis of their ability to render decisions 
quickly (often within seconds) and con- 
sistently with rigid platform-wide 
policies. They are not encouraged | 1 
to bring deliberative nuance or | 
contextual expertise to a deci- 
sion, or to question whether 
a policy that applies to bil- 
lions of people in hundreds 
of countries is adequate 
for the specific complaint they are 
viewing. It is their hidden emo- 
tional labour that keep platforms 
and automated moderation tools 
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afloat. Yet . 

these moderators remain profoundly 
undervalued, their welfare and expertise 
pushed to the side in the rush to moder- 
ate at scale. 





The rush towards automation also risks fur- 
ther obfuscating already opaque moder- 
ation processes, exacerbating platform 
transparency and accountability issues. As 
public reliance on platforms has grown, so 
too has their incontestable power to deter- 
mine what speech is acceptable online. 
Freedom of expression is increasingly reliant 
on secretive and unaccountable business 
practices. The classified nature of modera- 
tion policies makes it virtually impossible to 
assess whether automated tools are effective 
at keeping users safe. Platforms have been 
hesitant to make their policies public, and 
industry transparency reports and experi- 
ments like Facebook’s Oversight Board, an 
independent committee that reviews a tiny 
portion of Facebook’s moderation decisions 
(but importantly, not its content modera- 
tion policies or design features) offer narrow, 
self-selected forms of transparency. 


Automated moderation tools risk making this 


situation far worse. Their decisions are difficult to 
audit, assess, and understand even for developers, 
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let alone for regula- 

tors or third-party researchers who strug- 

gle to gain access to them. These tools are 

Mie often built to meet the needs of differ- 
iwas ent “governance stakeholders”, which 

Oe _ may not align with 
Wrestinntts . the interests of 

content moderation: technical * users or national 


and political challenges in 


the automation of platform : laws.’ To give one 


governance. Big Data - exampl e, the 
& Society, . : 

January— Syrian Archive, an 
eS open source initiative to document war 
DOI: crimes in the country, has routinely battled 

meeeen, YouTube’s algorithm to disable terrorist 

tne content which routinely fails to differ- 
entiate between videos glorifying vio- 

Mmepsnicina lence and those documenting abuses.° 

K. (2018, June = ae 
36) Youll Leaving thedecision , of what speech 
keeps deleting is allowed on the web to black 

of Sa box tools and black box policies, 
chemical with no independ- ent oversight 
Met or assessment, will only further 
iwivgee §=diminish the account- ability of large 
wae platforms and render the societal 

wired.co.uk/ ee a 

Mucem COst of automated tools invisible. 
chemical- 
weapons- 


frre Rather than a dream come true, the 
nn increasing reliance on automated 
Foca moderation tools risks becom- 
ing another AI night- 
mare. Platforms and 
regulators must not 


be constrained toa 
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narrative of “human vs. 
automated” modera- 
tion. Instead, they must 
ask a broader set of 
questions. For example, 
how can platforms be 
designed to encour- 
age safe behaviour? Is 
limiting their size and 
scale amore promising 
solution? Rather than 
following profit-driven 
metrics like maximis- 
ing user engagement 
or increasing content 
virality, what if plat- 
forms redesigned their 
affordances and fea- 
tures with safety front 
and centre? Rather 
than looping modera- 
tors in at the last min- 
ute to put out fires, 
/ what if firms included 
them in the design of 
these products 
from the out- 
set? 


And 
rather than exploit- 
ing outsourced mod- 
erators, further burying 
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their emotional labour, what if tech firms 
properly compensated and cared for their 
well-being as full-time employees? 


As with many other I technologies, the hype 
around automated moderation tools reflects 
a misplaced belief that complex sociopo- 

litical problems can be adequately resolved 
through technical solutions. AZ is tremendously 
helpful at addressing well-defined and con- 
crete tasks, but determining what speech is 
acceptable for billions of people is anything 
but a well-defined challenge. It is under- 

standable that we wish it were so—how 

nice would it be to simply let a machine 

police our speech for us without 

changing the scale, practices, 
or affordances of Facebook, 

Twitter or YouTube? 





Andrew 
Strait is a 
_ former Legal 
" Policy Specialist at 
Google and works on 
technology policy issues. 
He holds an MSc in Social 
Science of the Internet from 
the Oxford Internet Institute. 





‘ATION WON'T.. 





-Conso 


= i = 
ae fe 4 te lic “hm atin 
ei 





= ‘progress: 
pvt 


‘solutionism, 





and | 
farmer 
_ protests 
in India 


~~ Tulsi 
Parida 
Aparna | 
Ashdk  "34"F 


6i 
Ll 


162 


TECHNO-SOLUTIONISM AND JUGAAD 


In many low and middle-income econ- 
omies, the adoption of new technologies 
such as artificial intelligence is often 
done quickly and aggressively, with little 
oversight, in an effort to “catch up” with 
more advanced economies. The prevail- 
ing narrative is that countries still wad- 
ing through the debris of a colonial 
past need rapid techno- 
logical advancements 
for economic develop- 
ment. Recent digital 
policy consultations 
in countries such as 
India reflect an incli- 
nation to accelerate 
{l adoption even in the 
absence of essential 
regulatory protections. 
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1. https://niti.gov.in/ 





national-strategy- The Indian 
artificial-intelligence 2 

2018 National 

Strategy on 

Artificial Intel- ligence high- 


















lighted the “transformative impact” of 
Al and its potential to better the econ- 
omy, society, and environment.! As is the 
case with countries the world over, (Al is 
seen as a key component of devel- 

. 2. Tucker, I. 
opment. However, India BT March >) 
lacks regulatory meas- Evgeny Morozov: 

“We are abandoning all the checks 
ures to protect data po and balances.” The Guardian. 
vacy, which is essential https://www.theguardian.com/ 
to individual and con- technology/2013/mar/09/evgeny- 

. 7 morozoy-technology-solutionism- 

sumer rights. Privacy as (#ineweg 
a fundamental human 3. Saha, P. K. (2018, June 16) 
rig ht was established ‘Jugaad’ culture: The good, the 
in India only in 2017 bad and the ugly. Mint. https:// 
y ? www.livemint.com/Leisure/ 
and a personal data IZhswLKRpWI1QjnnAiGNNKM/ 

H ha Jugaad-culture-The-good-the-bad- 
protection bill is yet to iaee-ucly html 
be passed. Both gov- 
ernment and industry 
utilise a techno-solutionist narrative” 
to limited regulation and sometimes 
promote deregulation of sectors, which 
can result in substantial infractions of 


human rights and human development. 











In India, techno-solutionism is often 
paired with the concept of jugaad, the 
notion that low-cost innovations can 
solve local problems.’ Examples include 
everything from rejigging a truck so 
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it can power a rural 
village’s electric- 
ity, to Mangalyaan, 
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Mars orbiter mission 
achieved with a meagre budget and 
home-grown technologies. On the sur- 
face, jugaad seems an ingenious and agile 
approach to development. However, the 
jugaad mentality is eerily similar 
to Silicon Valley’s own “move fast © and 
break things”. While jugaad might have 
few consequences at a small scale, when 
applied to technologies such as Al that 
operate ata vast scale, without adequate 
oversight, this mentality can have dev- 
astating consequences. 
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Al AND POWER IMBALANCES 


“Al hype” for economic development 
is not unique to emerging econo- 

mies. The unique mix of jugaad and 
unregulated techno-solutionism 

in India, however, could result in 
rapid technological progress witha |) 
big down-side. When coupled with 
jugaad, unregulated techno-solu- 
tionism is dangerous in its belief 
that band-aid solutions and hastily 
created technologies are needed 
in the name of economic develop- 
ment. Without adequate protections, 
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=  thecountry’s ambitious Al adoption j[Rgpaeeuas 
an y 7 C., Garfinkel, B., 
= plans could result in catastrophic Leung, J. & Dafoe, A. 
ae . . i 
€ power imbalances‘ that come with SEyauaAMiias 
. Clause: Distributing 
a tremendous social cost. One of the [iictrenaes 
known potential risks of lisystemsis . mausa@aur 
7 Governance of AI 
the data-related power imbalances Research Report. 
that pit individuals against large con- | Rigeaaanieieen 
. nstitute, University 
glomerates that have substantial of Oxford. https:// 
— resources at their disposal. egy 1 1.0x.ac Uk 
a windfallclause/ 
Oo 
Pe i = 5. Cheney-Lippold, J. 
A Inan ideal world, the bene A. bac 
fits and risks of Al would New York, 


be distributed equita- . 
bly amongst firms and University 


society. However, Press. 
growing evidence 





shows a small 
number of firms have a 
substantial share of the 
Al market, and without con- 
sequential changes, those very 
firms will also have a disproportion- 
ate share of the benefits of 4’ while 
society bears the brunt of the risks.° 
Exploitative business models with an 
inequitable distribution of the risks 
and benefits of fl give rise to a con- 
solidation of power amongst those 
institutions that hoard data and use 
Al unchecked. 








E OF PROGRESS... 


However, when designed 
responsibly, in a context- 
appropriate manner, with 

adequate internal gov- 
ernance and external 
regulation, 
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AT technologies can 

play a role in solving 

problems at scale. For exam- 

ple, Al applied in diagnostics 

can compensate for the lack of 

skilled professionals needed to 

make healthcare accessible in 

rural parts of India. Especially for countries 
in the nascent stages of Al development, it 
is important to scrutinise proposed 
solutions with a socio-technical lens 
and critically evaluate the potential 
of power imbalances and foreseeable positive 
and negative consequences. 
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LOOKING AHEAD: AGRI-TECH AND FARMER 
PROTESTS 


LG 3.4835 


The combination of deregulation, the jugaad 
mindset, and techno-solutionist narratives & 
together create the ideal conditions for dispro- ) 
portionate power imbalances. A case in point is =. 
the “technological disruption” that is currently » 
underway in India’s agricultural sector. ; 


One of India’s largest conglomerates has 
recently ventured into agri-tech, claiming that 
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6. Kadidal, A. OU Al canimprove efficiencies inthe 
Mite ywentc. disorganised agricultural sec- 
invade agri to help 6 

India prosper: Ex tor.° In early 2020, the company 
eigen) announced an agri-tech plat- 

Miiteameuaene § form claiming to help farmers 
com/city/ai-must- J xa . 

invade-agri-to- make data-driven decisions in 

help-india-prosper- H H 

experts.791491 ial farming practices, and connect 
them directly to suppliers. With 

this proposed platform, this cor- 
poration, which also owns India’s 
largest supermarket chain and 
India’s largest telecom services, could 
conceivably gain complete control of 

supply and demand chains—with farmers 
relying on the app to sell their produce via 

the company’s telecom service directly to the 


company’s retail stores. 







Regulation to prevent such a consolidation of 
power is necessary to protect farmers’ rights. 
Instead, in late 2020, the Indian government 














a actively deregulated the sector by passing three 
Q heavily contested agricultural bills to liber- 
alise farming and “attract private sectors 
and foreign direct investment”.’ The new 
laws allow for (1) farmers to trade freely, 
(2) farmers to enter into contract farm- @aaaaiaee 
ing, and (3) deregulation of selected (@@Niteszanrs 
essential commodities, such as cere- OP Secemba 
als, edible oil, oilseeds, pulses, onion WZmrmensmauaie 
etc. The bills were passed hastily dur- ican at a 
% ing the Covid-19 pandemic and with- (Qanecstoeror 
& Outconsultation with farmers. eps-//pib-gov.in/ 


PressReleasePage 
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8. Damodaran, H. (2020, December 31) Explained: 






oe The concerns of farmers, and what Centre can negotiate 
pa to end protests. The Indian Express. 
° ! https://indianexpress .com/ 
ray article/explained/farmers-big- 
4 ! concern-and-what-govt-could- 
negotiate-7073291/ 
- 
= 
‘s 
s The new laws 
a4 . 
a have been crit- 
icised as “cor- porate-friendly and 
anti-farmer”, and this controversy 


has givenriseto oneof the largest labour 
movements in the world—a 
250 million-strong strike in 
support of farmer unions. The 
demands are clear. The three 
bills must be repealed. A mini- 
mum price and state procure- 
» ment of crops must be made a 
9. legal right.* Farmers have also 
eos made the connection between 


(2020, 
December the new laws and corporate 












ae power. They have called for 
umber of : 
Protesting a boycott of the company’s 


Farmers Switch 


i Jioto ' products, with many farmers 


porting their mobile service to 
other providers.’ 


Rival Mobile 
Networks. The 
Wire. https:// 
thewire.in/ ’ 
orcs: " This example of techno- 
jio-networks-airtel- solutionism in the agricul- 


pedatone-idea \ ture sector is emblematic 













































of the inequitable distribution of the risks 
and benefits of technology systems imposed 
on society. When the people who are being dis- 
enfranchised by il systems demand protection, 
whose concerns will be heeded? India strives to 
be arole model for countries in the region, and 
these choices could set a precedent. Gaining 
the trust of citizens through the equitable dis- 
tribution of risks and benefits would be helpful 
for \! adoption. At the time of writing, the out- 
come of the farmer protests is uncert2in—as 
are the effects that the laws and this movement 
will have on those most disadvantaged, namely 
landless Dalit farmers. Still, one thing is clear: 
farmers in the world’s largest democracy are 
ready to fight the imposition of emergent sys- 
tems of oppression. 


Tulsi Parida is a socio-technologist currently 
working on AI and data policy in fintech. Her 
previous work has been in edtech, with a focus 
on responsible and inclusive learning solutions. 


Aparna Ashok is an anthropologist, service 
designer, and AI ethics researcher. She spe- 
cialises in ethical design of automated deci- 
sion-making systems. 
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In Nigeria, Al is being 


touted by some as a one- 
size-fits-all solution to 
the country’s inefficien- 
cies and woes.! An unfor- 
tunate combination of 
sclerotic (and occasion- 
ally regressive) domes- 
tic development with 
Western-influenced 
tech solutionism has 
resulted in a burgeon- 
ing fascination with the 
technology. Nowhere, 
perhaps, is this more 
readily evident than in 
the fintech industry. 


1) Seb hvoyut-) i afoy-tve lm C4002 in 
March 8). Nigeria Needs 
Artificial Intelligence to 
Combat Insecurity, Says 
Expert. The Guardian 
Nigeria. https://guardian.ng/ 
news/nigeria-needs-artificial- 
intelligence-to-combat- 
insecurity-says-expert 
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2. Jackson, T. (2021, March) Nigerian Payments 
Startup Flutterwave Achieves ‘Unicorn’ Status after 
Hl $170m Funding Round. Disrupt Africa. 
Fintech, . https://disrupt-africa. 
a portman- com/2021/03/10/nigerian-payments- 
startup-flutterwave-achieves-unicorn- 
teau word that status-after-170m-funding-round/ 
describes the inte- 

. ' —_ 3. Kola-Oyeneyin, E., Kuyoro, M., 
gration of financial ser P & Olanrewaju, T. (2000, September 
vices with technology, IS 23). Harnessing Nigeria’s fintech 

= potential. McKinsey & Company. 
often deployed by com https://www.mckinsey.com/featured- 
panies offering financial insights/middle-east-and-africa/ 
services to penetrate harnessing-nigerias-fintech-potential 
new markets. Following | 
aseries of highprofile “* 
funding successes, espe- 
cially that of the “country’s lat- 
est unicorn”,” the term has become 


a buzzword in Nigeria. 
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According to a 2020 McKinsey report,’ 
Nigeria is home to over 200 standalone fintech 

companies. This figure does not include the 

dizzying array of fintech offerings by Nigerian 
brick-and-mortar banks. Yet, Nigeria is one of 
seven countries that contribute to nearly half 
of the world’s unbanked population, totalling 
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4. Ventura, L. 

(2021, February 17). Global 
Finance Magazine - 
World’s Most Unbanked 
Countries 2021. Global 
Finance Magazine. https:// 
www.gfmag.com/global- 
data/economic-data/ 
worlds-most- 
unbanked-countries 


5. Forty percent of 
Nigerians live below the 
poverty line: Report. 
(2020, May 4). AlJazeera. 
https://www.aljazeera.com/ 
economy/2020/5/4/forty- 
percent-of-nigerians-live- 
below-the-poverty-line-report 


6. Osakwe, S. (2021, April 6). 
How Is Nigeria’s National 
Financial Inclusion Strategy 
Going? Center for Financial 
W E/TonITRY (0)/ Ba v188 OSHA AYAY A 
centerforfinancialinclusion. 
org/how-is-nigerias- 
national-financial- 
inclusion-strategy-going 


about 60% of her adult popu- 
lation.‘ In part, this is due to 


the sheer number of peo- 
ple living in severe poverty 
(83 million compared to 
India’s 73 million).° This is 
exacerbated by a conserv- 
ative and seemingly erratic 
economic policy and a lack 
of access to financial ser- 
vices, particularly in rural 
areas.° With the median 
age of Nigerians being 
around 18 years, home- 
grown fintechs are step- 
ping up to the challenge 
of providing financial 
inclusion services to its 
youthful population.’ By 
using targeted advertise- 
ments, appealing offer- 
ings, and other digital 
marketing strategies, fin- 
techs may appear to have 
a better chance of reach- 
ing these young people 
than conventional banks. 


Certain Nigerian fintechs, 
which offer their services 
to individuals or 
corporate clients, claim to lever- 
age “machine learning algorithms” and 
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“Al-powered facial recognition” to 
assess credit risk, prevent fraud, 7. Roland Berger 
generate personality profile » @@pitiieiemucaam 
reports, and for identity veri- Financial Inclusion Strategy. 
fication.’ Several fintechs use Central Bank of Nigeria. https:// 
www.cbn.gov.ng/Out/2012/ 
these products. Acloser look publications/reports/dfd/CBN- 
at the environments in which aia realm 
these solutions are deployed (#i@ONfeeremtbnd 
raises serious questions yeu eaneuen 
about whether they can , November 19). The BackEnd: 
deliver on their promises | 7 iaiieaiain uta 
to improve financial inclu- ResnewWaiityiesste ce 
sion. Instead, the real risk analytics-intelligence-palantir-africa/ 
seems to be that these 
solutions could exclude 
minorities and privilege 
profit at the expense of 
individuals’ rights and lib- 
erties. To identify and pro- 
file the ideal customer, these 
companies typically access 
the personal data of private 
individuals from the govern- 
ment, through third-party service 
providers, or harvest it directly 
by sifting through personal infor- 
mation on mobile devices, such as 
text messages, fine and coarse loca- 
tion, media contents, contact lists, social 
media, and use of trackers and other per- 
missions. One company, which promises 
to empower Africans “by driving social 


and financial inclusion!” currently pays an 
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undisclosed amount to Nigerian gov- 
ernment agencies to gain access 
to millions of Nigerians’ national 
i , 5 . 
9, Hersey, F. (2020, identity data. The com 
August 5). Verify pany's publicly 


my life: could a Nigerian 
yeue)o) (001 
@@® lead toa global 
trust solution? (Or fuel 
a two-tier society?). 
Biometric Update. https:// 
www.biometricupdate. 
com/202008/verify-my-life- 
could-a-nigerian-problem-lead- 
to-a-global-trust-solution 















stated goal is 
to promote trust through digi- 
tal identity and verification services. 
Another company claims to help “banks 
distinguish between a photograph in an 
ID and a selfie”, to which end it has “cre- 

ated an identity management system that 

harnesses the powers of facial recogni- 
tion technology.” This system connects to 
government databases and non-govern- 
ment databases, such as that of the Nigeria ‘ 
Inter-Bank Settlement System (NIBSS). Both 
companies promise that /\l-powered facial rec- 
ognition systems will improve integrity and better 
identify fraud by “knowing all details about the 
client in near real-time.” 














The kind of access required by these systems 
is incredibly invasive. Ultimately amounting 
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_ to surveillance of the activities of pri- 
- vate citizens, it is potentially in breach 
of rights enshrined in the constitution 
and other laws, including freedom from 
discrimination, privacy, data protection 
and dignity. As other authors in this book 
have argued, these systems, which are not 
known for their accuracy or fciirness, could 
be unfairly prejudiced agcinst persons based 
on their gender, socio-economic background 
or other discriminatory factors. More troubling 
is the prospect of fintechs ending up as gate- 
keepers, shutting out the excluded groups they 
claim to include. If, for example, an individual 
is unable to purchase a smartphone (or elec- 
tricity or internet to make use of it) and one of 
these systems, therefore, cannot automatically 
extract data, it will rank them as undesirable, 
further excluding them from access to credit 
and financial products. 
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According to a report published by Tech Hive 
Advisory about the pervasive prac- _ 
tices of digital lenders in Nigeria,’” 
seven of the 22 mobile appli- Pach 
cations analysed publicly Hive Advisory 
' (CLUPATS sl-)oyutt-ta'9) 
disclose that they — A\ to Digital lending: Inside the 
determine borrowers id pervasive practice of LendTechs 
oo 4 ( in Nigeria. Linkedin. https://www. 
credit worthiness. Only e linkedin.com/posts/tech-hive- 
one mentioned the ex- * advisory_digital-lending-inside-the- 
' H H practice-of-lendtechs-activity- 
istence of profiling m % 676843 1134297620480-E5zx 
its privacy notice, as | 


is required by law."! 
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11. Article 3.1(7)(L) of the Nigeria Data 
Protection Regulation (NDPR). 
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Besides this disregard 
for the law, the report also 
notes an alarmingly com- 
mon lack of algorithmic 
transparency, lack of ex- 
plainability, lack of account- 
ability, and an absence of 
information on purpose 
limitation of the data used. 


It appears that the solutionism being sold by 
fintechs and other players like digital iden- 
tity providers, namely the belief that Al can sin- 
gle-handedly fix structural deficiencies, has 
been bought hook, line and sinker by those 
who ought to be the last to do so—the gov- 
ernment. Neglecting or outsourcing civic ob- 
ligations (such as digital identity registration, 
and in particular, the development of robust fi- 
nancial inclusion policies) to profit-driven pri- 
vate enterprises running machine learning 
algorithms without sufficient safeguards will 
undoubtedly worsen already-existing ine- 
qualities by systematically breaking down 
civil rights and freedoms. One such exam- 
ple is the licensing of private entities to ac- 
cess national identity biometric data held 
by the National Identity Management Com- 
mission, some of which cleim to use 
artificial intelligence and facial recog- 
nition.'* The detzils of the agreement 
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12. National Identity Management 
Commission (2020, 15 December) 
Public Notice: Approved Data 
Capturing Agents (Digital Identity 
Ecosystem) [Press release]. 
https://nimc.gov.ng/public-notice- 
approved-data-capturing-agents- 
digital-identity-ecosystem 
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are not 

available publicly. 
No record of a data 
protection impact 
assessment being 
conducted has 
either been pub- 
licly stated or pub- 
lished. These and 


other machine 
learning algorithms 


that make predic- 
tions based on his- 
torical events and 
data cannot be at the 
forefront of providing 
the forward-looking 
information we need 
for our future. 


While 4! has its uses 
in industry—fintechs 
included—it cannot 
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replace a com- 
prehensive financial 
' inclusion and develop- 
ment approach. Certainly not 
when spurred on by a lack of 
transparency and accounta- 
bility. The performance, val- 
idation and deployment of 
/\ must be ethical and meet 
existing legal requirements. 

F*rrness and transparency 

must determine limits to 

how data is used and the al- 
gorithms that are deployed. 
Security, privacy, data pro- 
tection, and accountability 
about how data is used and 
by whom is critical. Most importantly, it is es- 
sential to understand users, their needs, and 
the context in which these technologies 
are being used. 
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Europe has been lured in by the siren call 
of artificial intelligence. Public debate 
is characterised by snake oil promises 
of \\l’s benefits to the economy, social 
welfare, and urban development. Here, 
“Al” is a catch-all phrase used to describe 
a wide-ranging set of technologies, most of 
which apply statistical modelling to find pat- 
terns in large data sets and make predictions 


based on those patterns. Concerns rcised about 
the unpredictable nature and possible societal 
harms of Al models have given rise to a policy 


doctrine of ethical and procedural safeguards, 


the idea being that A\’s “great” potential can be 
harnessed and its harms mitigated by imple- 
menting safeguarding principles of non-bind- 
ing fairness, accountability, and transparency. 
Building on our work as researchers and practi- 
tioners in the field of technology and society, we 
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will discuss 

one of these 

safeguards, 
namely algo- 











where and how A! is 
used in a city. Extolled by 
some in the AI ethics commu- 
nity as an example of good AI governance, 
we argue that voluntary ethical and pro- 
cedural A\ guardrails in fact perpetuate the 
hype and neutralise important critical debate. 


ALGORITHMIC REGISTERS IN EUROPE 


In line with these ethical activities, a 
number of European cities! are experi- 
menting with algorithmic registers run 
by local municipalities. In September 
2020, the cities of Amsterdam and 
Helsinki launched their registers to 
ae increase transparency around the 
. Johnson, K. . 3 
Cues deployment of /\l in the public 
eee ae eel sector. These databases col- 
elsinki launch . F 
algorithm registries lect information about how A 
fo bing uauspae nC systems are used in an open 
public deployments of . 
AI. Venturebeat. https:// and voluntary fashion, 
CH OATTS OE ean AU) which should provide 
amsterdam-and-helsinki- "alee = 
launch-algorithm-registries-to- insights into the local 
bring-transparency-to-public- uses of these systems 
deployments-of-ai/ . ys . 
Both the Helsinki and 


















the Amsterdam register contain only five 
entries. The entries function on an opt-in 
basis and mostly cover automated govern- 
ment services, including city libraries, hospi- 
tals, crowd management and parking control. 
Alittle over two weeks after the launch of these 
registers in the autumn of 2020, prominent Al 
ethics scholar Luciano Floridi published an edi- 
torial letter in Philosophy & Technology,’ in which 
he heralded them as solutions for the many chal- 
lenges of Al governance, not least those related 
to public accountability and trust in Al. 





There are a number of governance assump- 
tions attributed to the registers that we seek to 
question, especially regarding the ex-post, or 
after the fact, framework of “accountability 
through transparency” contained within 


2. Floridi, L. the register concept. Some of the most 
(CLOW Nuititer.| 
Intelligence as a 
Public Service: Learning from 
Amsterdam and Helsinki. Philosophy =e oe 
& Technology, 33(4), 541-546. DOI: 
10.1007/s13347-020-00434-3 


3. Buolamwini, J., & Gebru, T. (2018). 
Gender shades: Intersectional accuracy 
disparities in commercial gender classification. 
Conference on Fairness, Accountability, 
and Transparency (pp. 77-91) 
http://proceedings.mlr.press/v8 1/ 
buolamwinil 8a/buolamwini18a.pdf; 
PaNaavi ejay bvlccsunt-le(oyit-) mM @40P40)) 
Netherlands: We sense trouble: 
Automated discrimination and 
mass surveillance in predictive policing 
in the Netherlands. 
https://www.amnesty.org/en/ 
documents/eur35/297 1/2020/en/ 
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harmful Al applications 
are evidently missing 
from the Amsterdam 
and Helsinki regis- 
ters. The databases 
contain no mention 
of known welfare 
and law enforce- 
ment applications. 
For example, 
the Amsterdam 
object detec- 
tion entry, in 
which the 
city is 
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experimenting with GAIT 
recognition for crowd mon- 
itoring purposes, does not 
account for the police facial rec- 
ognition trials taking place in these 
same locations. This means that some of 
the most sensitive applications of Al, often 
implicated in algorithmic discrimination,’ are 
not currently covered by the registers, and it 
is unclear if they will be in the future. 
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The lack of critical engagement by AJ propo- 
nents with these information voids in the reg- 
isters is telling of their inability to function as a 
transparency tool between the city and its res- 
idents. By defining accountability as transpar- 
ency through voluntary registration, proponents 
of algorithmic registers are essentially taking 
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an ex-post approach to .'y 
governance too: push Ii 
systems as a public utility first 
(“Just do it”) and ask for forgiveness 
later. This reinforces the assump- 
tions that /\ is neutral and should be 
used “for the greater good”, and neu- 
tralises criticism as simply a matter 
of imperfect information. This gov- 
ernance-by-database eschews 
difficult conversations 
about whether AI 
systems should 

be implemented 
at all, and how 


























these systems 
are advancing puni- 
tive politics that pri- 
marily target already 
vulnerable populations. 






PERPETUATING THE Al HYPE 


What is even more telling than the 
registers themselves, however, is the 
lack of critical engagement by Al pro- 
ponents with the power structures 
and political ideologies that shape 
these governance-by-database solu- 
tions. Isolating governance mecha- 
nisms outside their social, political, 
and economic context allows for 
the perpetuation of a discourse that 
reaffirms the arbitrary notion of “Bl 
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for good”. This ignores 
the fact that most 
algorithms in use for 
urban management 
pre-date the idea of 
registers and are deeply 
rooted ina politicalide- 
ology and organisa- 
tional culture bent towards 
efficiency and cost reduc- 
tion. Efforts to abstract and 
generalise Al accountability 
frameworks allow their pro- 
ponents to move beyond 
the messy nature of 
reality and to further 
depoliticise 4; by replac- - © 
ing the outdated idea 
that technology is “neutral” with the notion 
that the “great” potential of /\ can be har- 
nessed when harms are mitigated through 
voluntary procedural and ethical safeguards. 
Lauding the registers without understanding 
their context discounts concerns about the 
negative impact of i on society, because it is 
this which aligns safeguards with the political 
environment and commercial interests that are 
enabling the Mi hype. 





The deployment of AV for public services, from 
the critical (like urban infrastructure, law en- 
forcement, banking, healthcare, and human- 
itarian cid) to the more mundane (like parking 
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control and libraries), should be done 
with great caution. Not least as these 
systems are often implemented on top 
of, and in line with, existing neoliberal 
politics that are punitive and extractive. 
As such, Al systems cannot be 
seen as isolated from the 
context in which they 
are deployed. When 
looking at these reg- 
isters, and other opt-in 
accountability frameworks, 
we need to consider what is 
missing and who is steering the 
conversation. Due to the complex 
nature of governance, registers are 
only a partial and, at times, overhyped 
answer to ensuring public accounta- 
bility for the application of AZ systems. 
Indeed, the ex-post model bypasses 
critical conversations about the root 
motivations for rolling out &, systems, 
and who is truly served by them. 
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Around the same time that Turing was devising 
his famous test, many countries were pouring 
concrete into the foundations of the world’s 
first nuclear power stations. Nuclear power 
then was a technology that, like AJ today, gen- 
erated as much hope and hype as it did anxiety. 


Today, the abundant hype around Al is being 
met with growing resistance towards harmful 
algorithms and irresponsible data practices. 
Researchers are building mountains of evi- 
dence on the harms that flawed /] tools may 
cause if left unchecked. Civil rights campaign- 
ers are winning cases against AI-driven sys- 
tems. Teenagers scorned by biased grading 
software are carrying placards and shouting 
“F*** the Algorithm!”. Public trust in Al is on 
rocky ground. 


THE POWER OF RESISTANC 
























This puts Al’s current success on a poten- 
tial tipping point. Fuelled by misleading 
Al “snake oil”, today’s unrealised hype 
and very real harms are creating public 
resistance that could push Al into another 
stagnant winter that stifles both its harms 
and its benefits—much like those of the 
1970s and 80s, when Al innovation was 
curbed and political will failed. But resist- 
ance also highlights what responsible, 
. publicly acceptable technology prac- 
\ tice could look like, and could help 
prioritise people and society over 
profit and power. To understand 
how, nuclear power’s history of 
» hype and resistance offers 
a useful guide. After all, 
the term “Al winter” was 
in part inspired by the idea 
1. Crevier, D. (1993) of nuclear winter.’ 
Al: The Tumultuous 
Search for Artificial 
Intelligence. New Despite the lethal dangers 
Bock, aa of nuclear technology being 
made horrifyingly evident at 
Nagasaki and Hiroshima in the 
closing days of the Second World 
War, the following years saw 
nuclear’s military applications 
moulded to civil purposes. In the 
early 1950s, its potential for good 
was manifested in the world’s very first 
nuclear power station at Calder Hall in 
Cumbria, UK, now known as Sellafield. 
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Since then, trends in public acceptance 

of nuclear power have been well-studied, 

and thousands of people across the world 

have been surveyed. These studies have 

shown how, in nuclear power’s early 

years, the fearsome image of mush- 4) 
room clouds was perceived as 
the distant past, while civil 
nuclear power was promised 
by governments and the industry as not 
only the future of electricity generation, 








but as the herald of a brave, new, tech- p. 
nological society built on unlimited = 
energy.’ However, many hyped-up Modigliani, 


promises never materialised (such (igawaad 
as Ford’s nuclear-powered vehi- Soyiteqneant 
cle, the Nucleon) and the fear ae 
of nuclear war lingered. Power: A 
(oy irarastoyi 
. Approach. 
Then, in 1986, the potent, danger- (Wiad 
ous reality of nuclear power was 'WAgag 
Sociology,95 
reproven when reactor No.4 at the NGA 


Chernobyl nuclear power station 
failed catastrophically. The combined 
technical, chemical, institutional, and admin- 
istrative break-down killed thousands according 
to official estimates, and spewed radiation across 
thousands of square miles (reaching as far as 
Sellafield in Cumbria, in a bitter irony). 


In the wake of the Chernobyl 
disaster, support for— 
and investment 
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in—civil nuclear power plummeted. Fears of 
nuclear’s destructive force were absorbed 
into the growing environmental movement, 
and the early enthusiasm for nuclear power 
swiftly gave way to a “dark age” for the tech- 
nology, in which resistance grew and innovation 
stumbled. Nuclear power saw little new activ- 
ity until the mid-2000s, when some nations, 
including the UK, France, US and China, began 
— — (and continue) 
=e to reinvestin | 
nuclear | 
power, 
in 
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part thanks to public recognition that 
__/ it generates low-carbon electricity. 





Though the public is often dismissed as 
having little influence on the material shape of 
technology, the history of nuclear power shows 
that public opinion can have a profound impact 
on policy and practice—that is, how governments 
invest in and regulate technologies, how innova- 
tors develop them, and how people use them. 
This is seen with many other technologies too, 
from genetic engineering to cars, and now in ll. 
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The recent kindling of public resistance agzinst 

Al must be recognised as the vital signal that it 

is. Previous Al winters of the late 20th Century 
followed seasons of over-promising: while 
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computing hardware was catching up with Al 
theory, excitement waned as the pace of inno- 
vation slowed and hyped-up claims about what 
Al could do were left unfulfilled. Today, however, 
superfast processors, networked cloud servers, 
and trillions of bytes of data allow many of those 
cleims to be apparently realised. Al systems are 
helping detect and diagnose macular degen- 
eration, powering GPS systems, and recom- 
mending the next best thing since Tiger King. 









But success 
is a double-edged sword. 
As Al’s abilities are put to use in 
everyday applications, so too are its 
dangers. The dubious data prac- 
tices of firms like Cambridge 
Analytica have disrupted 
politics, algorithms like () 
COMPASS have exacer-__, 
bated pre-existing bias and 
injustice, and facial recogni- 
tionsystemsdeployed _ 
by law enforcement are | 
infringing on civiland ~— 
data rights. As peo- 
ple realise they are largely 
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powerless to challenge and change these 
systems by themselves, public concern 

and resistance has been spreading to all 

corners of the Al landscape, led by civil 
rights campe@gners, researchers, activists, 
and protesters. If ignored, this resistance 
won’t just weed out harmful Al. It may sti- 

fle the social and political will that supports 
beneficial AT too. 






Social scientists remind us that correlation is 
not causation. Simply because public support 
and resistance to nuclear power has neatly 
tracked levels of government investment and 
technological innovation does not mean one 
creates the other. But an established body of 
research shows that they are connected, and 
that public resistance is an important signal. 


After Chernobyl, public resistance to nuclear 

power made clear that its risks and dangers 
were no longer considered acceptable. This had 
a chilling effect on political will and investment. 
It wasn’t until industry and governments com- 
mitted to more responsible, transparent, and 
accountable practices that investment and inno- 
vation in nuclear power began to reignite. Even 
today, nuclear power is not a widely accepted 
technology—public support is rocked still by 
incidents like the 2011 Fukushima disaster—and 
healthy public scepticism plays an active part in 
balancing hype against responsible practice. 
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Silicon 
chips and 
plutonium 
fuel rods do not share 
much in common techni- 
cally, and the threat of 
’ nuclear meltdown dif- 
fers from the threats 
posed by biased algo- 
rithms. But we can- 
not let the hope that 
‘J may never have a 
disaster on the same 
scale as Chernobyl lure 
us into complacency. 
The many small, individual 
disasters that are already 
occurring every day around us 
will continue to add up. As irre- 
sponsible i practices lead to 
social, political, cultural, and eco- 
nomic harms, public acceptance will 
falter and resistance will grow. 
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However, resistance is not a force to fear: 
it is a powerful signal. It may threaten 
the current hype-fuelled 4; summer, but 
it need not stifle responsible innova- 

tion. Harnessed well, public resistance 
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can help 

shine light 

on what must be 
improved, weed out, Al 
“snake oil”, define what 

is socially acceptable, and 
help a more responsible ii 
industry flourish. But if it is 
ignored, the current public 
mood around algorithms 
and big data could fore- 
cast more than just the 
winds of change. It could 
be the first cold breeze of 
another i! winter. 
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DESIGN NOTES: AlL-constrained Design 


The design of this book explores the use of artificial intelligence 
to conceptualise its main visual elements. From an optimistic 
standpoint of AI abundance, the book design is a semi-fiction, a 
staged and micromanaged use of a GAN (Generative Adversarial 
Network), an unsupervised machine learning framework, where 
two neural networks contest with each other in order to generate 
visual output. Stretching the narrative, this book could be framed 
as a/the (first) book designed by an AI. In this scenario, the collab- 
orating AI (more like the Al-as-head-of-design-that-doesn t-know- 
how-to-design), has informed, but also constrained the possibilities 
to work visually with the pages. 


The design strategy adopts the Wizard of Oz Technique , a method 
originated from interaction design where what is seemingly auton- 
omous, is in reality disguising the work of humans ‘as a proxy 
for the system behind the scenes’'. The use of the GAN, which a 
reader could expect as a simplification, a symbol of technological 
ergonomics, has instead complicated the process. As a result, the 
contents contort around the spaces that the AI imagination left 
them to exist, revealing an apparently spontaneous visual language. 


The book features results from two separate datasets, addressing 
the overall layout composition, and a (overly sensitive) recognition 
algorithm which targets all instances of ‘AI, ai, Ai’, regardless of 
their position or meaning. 


MetaGAN v.3 Layouts 


The dataset used to produce the compositions above is a collec- 
tion of book scans. The purpose of an image GAN is to create 
new instances by detecting, deconstructing and subsequently 
reconstructing existing patterns to create speculations about con- 
tinuations. Reusing existing layout materials ,conceived by human 
creativity, opens up the discussion of AI creativity. The outcomes, 
which could be perceived as surprising, original and novel, are 
however subject to human selection and valuation. In training the 
MetaGAN, the dissimilarity of the data points, in combination 
with the small size of the dataset (200 images), led to the idio- 
syncrasy of overfitting. An overfitted model generates outcomes 
‘that correspond too closely or exactly to a particular set of data, 
and may therefore fail to fit additional data or predict future obser- 
vations reliably”. 
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Al type results 


These AI letterings are results of a GAN using a dataset containing 
logos from various AI related brands (or belonging to Anguilla, 
whose country code top-level domain is ‘.ai’). The use of these 
characters is indeed automated in the design of the book, but it is 
done using GREP styles. 
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From predicting criminality to sexual orientation, 
fake and deeply flawed Artificial Intelligence (AI) 
is rampant. Amidst this feverishly hyped atmos- 
phere, this book interrogates the rise and fall of 
Al hype, pseudoscience and snake oil. Bringing 
together different perspectives and voices from 
across disciplines and countries, it draws con- 
nections between injustices inflicted by inappro- 
priate Al. Each chapter unpacks lazy and harmful 
Te made by developers when designing 
Al tools and systems, and examines the existential 
underpinnings of the technology itself to ask: why 
are there so many useless, and even dangerously 
flawed, M systems? 
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